2024-09-10T20:09:02.121 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T20:09:02.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:09:02.214 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899557 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: skanta@redhat.com first_in_suite: false job_id: '7899557' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAyQQwN6D0jKEg4O0aB8skOTWzKZG6G/8kmaHDZyPUifHph+LPmWDM7H1Yo/VwzMjozYbdbnaz5RwG7jYZ/FKrI= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T20:09:02.214 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T20:09:02.215 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T20:09:02.215 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T20:09:02.217 INFO:teuthology.task.internal:Checking packages... 2024-09-10T20:09:02.239 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T20:09:02.239 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T20:09:02.239 INFO:teuthology.packaging:ref: None 2024-09-10T20:09:02.239 INFO:teuthology.packaging:tag: None 2024-09-10T20:09:02.239 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:09:02.239 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:09:02.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:09:02.382 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T20:09:02.384 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T20:09:02.389 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T20:09:02.389 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T20:09:02.393 INFO:teuthology.task.internal:Saving configuration 2024-09-10T20:09:02.402 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T20:09:02.407 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T20:09:02.424 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:04:12.995026', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAyQQwN6D0jKEg4O0aB8skOTWzKZG6G/8kmaHDZyPUifHph+LPmWDM7H1Yo/VwzMjozYbdbnaz5RwG7jYZ/FKrI='} 2024-09-10T20:09:02.424 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T20:09:02.429 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-10T20:09:02.430 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T20:09:02.482 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdf3c393be0>, signals=[15]) 2024-09-10T20:09:02.482 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T20:09:02.488 INFO:teuthology.task.internal:Opening connections... 2024-09-10T20:09:02.488 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-10T20:09:02.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:09:02.566 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T20:09:02.576 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-10T20:09:02.654 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-10T20:09:02.655 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-10T20:09:02.712 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T20:09:02.713 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T20:09:02.714 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-10T20:09:02.735 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T20:09:02.743 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T20:09:02.750 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T20:09:02.750 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:09:02.769 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T20:09:02.776 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T20:09:02.776 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:09:02.838 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T20:09:02.871 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T20:09:02.886 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T20:09:02.887 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T20:09:02.887 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:09:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:09:02.905 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-10T20:09:02.963 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-10T20:09:02.963 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-10T20:09:02.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-10T20:09:08.577 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.2 MB 00:03 2024-09-10T20:09:12.403 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 20 MB 00:02 2024-09-10T20:09:16.695 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-10T20:09:18.187 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-10T20:09:19.500 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-10T20:09:24.930 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T20:09:26.912 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T20:09:26.912 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T20:09:26.998 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T20:09:27.006 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:09:27.007 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-10T20:09:27.008 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-10T20:09:27.008 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:09:27.107 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 308 kB/s | 23 kB 00:00 2024-09-10T20:09:29.257 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 7.0 MB/s | 16 MB 00:02 2024-09-10T20:09:31.165 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 8.4 MB/s | 35 MB 00:04 2024-09-10T20:09:31.424 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.6 MB/s | 29 MB 00:04 2024-09-10T20:09:31.424 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:09:31.424 INFO:teuthology.orchestra.run.smithi181.stdout:Total 18 MB/s | 79 MB 00:04 2024-09-10T20:09:32.105 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:09:32.446 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:09:32.447 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:09:33.411 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:09:33.411 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:09:35.860 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:09:36.490 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:09:36.582 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:09:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:09:39.503 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:09:44.464 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:09:44.503 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:09:49.359 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:10:27.783 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:10:27.823 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:10:28.155 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:10:28.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T20:10:28.155 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T20:10:28.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T20:10:29.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:10:29.002 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:10:29.364 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:10:29.449 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:10:29.449 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T20:10:29.449 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:10:29.449 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:10:29.450 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T20:10:29.450 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:10:29.450 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T20:10:29.508 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T20:10:29.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T20:10:29.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-10T20:10:29.548 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T20:10:29.549 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:10:29.550 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T20:10:29.551 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T20:10:29.551 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-505.el9.x86_64 2024-09-10T20:10:29.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T20:10:30.145 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-10T20:10:31.444 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T20:10:31.479 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-10T20:10:31.492 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-10T20:10:31.508 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.e8vfHNwSwU 2024-09-10T20:10:31.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.e8vfHNwSwU 2024-09-10T20:10:31.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.e8vfHNwSwU 2024-09-10T20:10:31.707 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.e8vfHNwSwU is 6KB 2024-09-10T20:10:31.727 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.e8vfHNwSwU 2024-09-10T20:10:31.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T20:10:31.815 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T20:10:31.815 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T20:10:31.815 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T20:10:31.815 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T20:10:31.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T20:10:31.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-10T20:11:01.993 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T20:11:01.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:11:01.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:11:20.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-10T20:11:29.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:11:29.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:11:32.605 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-10T20:11:44.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:11:44.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:11:47.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-10T20:12:02.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:12:02.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:12:05.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-10T20:12:23.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:12:23.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:12:24.199 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-10T20:12:24.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:12:24.584 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T20:12:24.584 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-10T20:12:24.600 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-505.el9.x86_64 2024-09-10T20:12:24.600 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T20:12:24.601 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:12:24.601 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T20:12:25.601 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T20:12:25.602 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T20:12:25.961 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-10T20:12:25.983 DEBUG:teuthology.parallel:result is None 2024-09-10T20:12:25.984 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T20:12:25.997 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T20:12:25.997 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:12:26.015 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T20:12:26.024 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T20:12:26.033 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T20:12:26.033 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:12:26.102 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T20:12:26.110 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T20:12:26.111 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:12:26.176 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:26.187 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:26.188 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T20:12:26.196 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T20:12:26.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:12:26.255 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T20:12:26.264 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T20:12:26.265 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:12:26.311 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:12:26.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:12:26.487 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:12:26.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:12:26.569 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:12:26.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:12:26.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-10T20:12:26.711 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:12:27.101 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T20:12:27.108 INFO:teuthology.task.internal:Starting timer... 2024-09-10T20:12:27.108 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T20:12:27.121 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T20:12:27.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-10T20:12:27.197 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-10T20:12:27.199 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T20:12:27.199 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-10T20:12:27.223 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-10T20:12:27.223 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-10T20:12:27.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:12:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:12:27.289 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T20:12:27.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-10T20:12:27.364 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T20:12:27.372 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T20:12:27.387 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T20:12:27.388 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T20:16:09.441 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-10T20:16:09.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:16:09.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:16:09.524 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-10T20:16:09.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-10T20:16:09.613 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T20:16:09.624 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T20:16:09.624 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:16:09.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:16:09.682 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T20:16:09.695 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T20:16:09.720 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-10T20:16:09.732 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-10T20:16:09.746 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T20:16:09.759 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T20:16:09.820 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:16:09.824 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:16:09.824 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-10T20:16:09.824 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:16:09.824 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:16:09.824 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:16:09.825 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:16:09.825 INFO:teuthology.run_tasks:Running task install... 2024-09-10T20:16:09.833 DEBUG:teuthology.task.install:project ceph 2024-09-10T20:16:09.833 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T20:16:09.833 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T20:16:09.834 INFO:teuthology.task.install:Using flavor: default 2024-09-10T20:16:09.839 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T20:16:09.839 INFO:teuthology.task.install:extra packages: [] 2024-09-10T20:16:09.839 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T20:16:09.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:16:09.991 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T20:16:09.992 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T20:16:10.106 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T20:16:10.106 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:16:10.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T20:16:10.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T20:16:10.133 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T20:16:10.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T20:16:10.276 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T20:16:10.340 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-10T20:16:10.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-10T20:16:10.566 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-10T20:16:10.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-10T20:16:11.154 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 304 kB/s | 84 kB 00:00 2024-09-10T20:16:11.655 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 52 kB/s | 24 kB 00:00 2024-09-10T20:16:11.988 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-09-10T20:16:12.697 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-10T20:16:14.557 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-10T20:16:19.933 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-10T20:16:21.491 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-10T20:16:22.912 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-10T20:16:28.343 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T20:16:30.720 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:16:30.733 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T20:16:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T20:16:30.735 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T20:16:30.736 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout:Install 31 Packages 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-10T20:16:30.737 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:30.738 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 62 M 2024-09-10T20:16:30.738 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:16:31.678 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 7.3 MB/s | 5.5 MB 00:00 2024-09-10T20:16:31.811 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 12 MB/s | 11 MB 00:00 2024-09-10T20:16:31.836 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 159 kB/s | 25 kB 00:00 2024-09-10T20:16:31.870 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 13 MB/s | 755 kB 00:00 2024-09-10T20:16:31.895 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 8.5 MB/s | 509 kB 00:00 2024-09-10T20:16:31.962 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 676 kB/s | 45 kB 00:00 2024-09-10T20:16:32.171 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): ceph-common-19.3.0-4858.gbad1f558.el9.x 17 MB/s | 22 MB 00:01 2024-09-10T20:16:32.196 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-ceph-common-19.3.0-4858.gbad1f5 613 kB/s | 143 kB 00:00 2024-09-10T20:16:32.221 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 3.1 MB/s | 161 kB 00:00 2024-09-10T20:16:32.246 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 6.2 MB/s | 321 kB 00:00 2024-09-10T20:16:32.272 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 6.0 MB/s | 306 kB 00:00 2024-09-10T20:16:32.297 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T20:16:32.372 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-10T20:16:32.489 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 890 kB/s | 171 kB 00:00 2024-09-10T20:16:32.515 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-10T20:16:32.540 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): boost-program-options-1.75.0-8.el9.x86 399 kB/s | 107 kB 00:00 2024-09-10T20:16:32.573 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 540 kB/s | 45 kB 00:00 2024-09-10T20:16:32.632 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-10T20:16:32.657 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-10T20:16:32.682 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 388 kB/s | 42 kB 00:00 2024-09-10T20:16:32.717 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-10T20:16:32.750 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 388 kB/s | 36 kB 00:00 2024-09-10T20:16:32.775 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 513 kB/s | 47 kB 00:00 2024-09-10T20:16:32.825 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-10T20:16:32.859 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 342 kB/s | 26 kB 00:00 2024-09-10T20:16:32.884 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 855 kB/s | 49 kB 00:00 2024-09-10T20:16:32.910 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T20:16:32.952 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-10T20:16:32.994 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-10T20:16:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-10T20:16:33.211 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): librados2-19.3.0-4858.gbad1f558.el9.x8 16 MB/s | 3.5 MB 00:00 2024-09-10T20:16:33.286 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-09-10T20:16:33.396 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-10T20:16:33.401 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:16:33.401 INFO:teuthology.orchestra.run.smithi181.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-10T20:16:33.513 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:16:33.550 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:16:33.551 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:16:34.216 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:16:34.216 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:16:35.713 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:16:35.795 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T20:16:35.825 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T20:16:35.850 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T20:16:35.881 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T20:16:35.908 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T20:16:36.099 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T20:16:36.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T20:16:36.539 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T20:16:36.615 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:36.711 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:36.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T20:16:36.828 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T20:16:36.891 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T20:16:36.928 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T20:16:36.972 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T20:16:37.041 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T20:16:37.063 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T20:16:37.119 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T20:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T20:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T20:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T20:16:37.338 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T20:16:37.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T20:16:37.407 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T20:16:37.439 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T20:16:37.494 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T20:16:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T20:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T20:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T20:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T20:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T20:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T20:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T20:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T20:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T20:16:38.708 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T20:16:40.952 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:40.996 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:41.580 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T20:16:41.617 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T20:16:41.707 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T20:16:41.707 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T20:16:41.707 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T20:16:41.707 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:41.742 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:51.563 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:51.563 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-10T20:16:51.563 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-10T20:16:51.563 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-10T20:16:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-10T20:16:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-10T20:16:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-10T20:16:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-10T20:16:51.564 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:52.610 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T20:16:52.636 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T20:16:52.636 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:16:52.636 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T20:16:52.636 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:16:52.637 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:16:52.637 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:52.638 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T20:16:52.721 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T20:16:52.721 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:53.901 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:53.901 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T20:16:53.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T20:16:53.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T20:16:53.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T20:16:53.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T20:16:53.904 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T20:16:53.905 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T20:16:53.905 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T20:16:53.905 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T20:16:53.905 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T20:16:54.843 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T20:16:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.845 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:54.846 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:16:55.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-10T20:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:27 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-10T20:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 50 M 2024-09-10T20:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 215 M 2024-09-10T20:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:16:56.142 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 759 kB/s | 161 kB 00:00 2024-09-10T20:16:56.517 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-10T20:16:56.568 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 477 kB/s | 304 kB 00:00 2024-09-10T20:16:57.443 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-10T20:16:57.445 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:16:57.445 INFO:teuthology.orchestra.run.smithi181.stdout:Total 33 MB/s | 50 MB 00:01 2024-09-10T20:16:57.473 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:16:57.493 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:16:57.493 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:16:57.871 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:16:57.872 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:16:58.407 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:16:58.487 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T20:16:58.525 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T20:16:58.553 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T20:17:03.252 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T20:17:03.335 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T20:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T20:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T20:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T20:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T20:17:06.075 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T20:17:06.075 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:06.075 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:06.075 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:06.076 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:06.076 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T20:17:06.076 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T20:17:06.076 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:06.076 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:06.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-10T20:17:06.960 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:38 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:07.103 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:07.105 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:07.105 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:07.105 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:07.105 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:07.105 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T20:17:07.106 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T20:17:07.107 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:07.108 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout:Install 37 Packages 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 M 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 109 M 2024-09-10T20:17:07.109 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:07.710 INFO:teuthology.orchestra.run.smithi181.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-10T20:17:07.893 INFO:teuthology.orchestra.run.smithi181.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-10T20:17:07.969 INFO:teuthology.orchestra.run.smithi181.stdout:(3/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-10T20:17:08.061 INFO:teuthology.orchestra.run.smithi181.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 3.1 MB/s | 291 kB 00:00 2024-09-10T20:17:08.153 INFO:teuthology.orchestra.run.smithi181.stdout:(5/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-10T20:17:08.278 INFO:teuthology.orchestra.run.smithi181.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-10T20:17:08.320 INFO:teuthology.orchestra.run.smithi181.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-10T20:17:08.362 INFO:teuthology.orchestra.run.smithi181.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-10T20:17:08.420 INFO:teuthology.orchestra.run.smithi181.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-09-10T20:17:08.446 INFO:teuthology.orchestra.run.smithi181.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-09-10T20:17:08.471 INFO:teuthology.orchestra.run.smithi181.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-10T20:17:08.763 INFO:teuthology.orchestra.run.smithi181.stdout:(12/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86 20 MB/s | 17 MB 00:00 2024-09-10T20:17:08.805 INFO:teuthology.orchestra.run.smithi181.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 525 kB/s | 188 kB 00:00 2024-09-10T20:17:08.830 INFO:teuthology.orchestra.run.smithi181.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 480 kB/s | 172 kB 00:00 2024-09-10T20:17:08.847 INFO:teuthology.orchestra.run.smithi181.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-10T20:17:08.872 INFO:teuthology.orchestra.run.smithi181.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-10T20:17:08.889 INFO:teuthology.orchestra.run.smithi181.stdout:(17/37): python3-markupsafe-1.1.1-12.el9.x86_64 276 kB/s | 35 kB 00:00 2024-09-10T20:17:08.906 INFO:teuthology.orchestra.run.smithi181.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 708 kB/s | 42 kB 00:00 2024-09-10T20:17:08.923 INFO:teuthology.orchestra.run.smithi181.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 439 kB/s | 22 kB 00:00 2024-09-10T20:17:08.974 INFO:teuthology.orchestra.run.smithi181.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-10T20:17:08.991 INFO:teuthology.orchestra.run.smithi181.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 516 kB/s | 43 kB 00:00 2024-09-10T20:17:09.016 INFO:teuthology.orchestra.run.smithi181.stdout:(22/37): python3-cherrypy-18.6.1-2.el9.noarch.r 8.8 MB/s | 358 kB 00:00 2024-09-10T20:17:09.041 INFO:teuthology.orchestra.run.smithi181.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-09-10T20:17:09.066 INFO:teuthology.orchestra.run.smithi181.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-10T20:17:09.091 INFO:teuthology.orchestra.run.smithi181.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-10T20:17:09.117 INFO:teuthology.orchestra.run.smithi181.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-09-10T20:17:09.142 INFO:teuthology.orchestra.run.smithi181.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-10T20:17:09.167 INFO:teuthology.orchestra.run.smithi181.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-10T20:17:09.193 INFO:teuthology.orchestra.run.smithi181.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-10T20:17:09.218 INFO:teuthology.orchestra.run.smithi181.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-10T20:17:09.243 INFO:teuthology.orchestra.run.smithi181.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T20:17:09.268 INFO:teuthology.orchestra.run.smithi181.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-10T20:17:09.294 INFO:teuthology.orchestra.run.smithi181.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-10T20:17:09.319 INFO:teuthology.orchestra.run.smithi181.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T20:17:09.344 INFO:teuthology.orchestra.run.smithi181.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-10T20:17:09.369 INFO:teuthology.orchestra.run.smithi181.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-10T20:17:09.403 INFO:teuthology.orchestra.run.smithi181.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T20:17:09.407 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:09.407 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-10T20:17:09.554 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:09.596 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:09.596 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:10.088 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:10.088 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:11.447 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:11.514 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T20:17:11.547 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T20:17:11.575 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T20:17:11.645 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T20:17:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T20:17:11.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T20:17:11.874 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T20:17:11.985 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T20:17:12.034 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T20:17:12.069 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T20:17:12.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T20:17:12.134 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T20:17:12.167 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T20:17:12.208 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T20:17:12.252 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T20:17:12.285 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T20:17:12.333 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T20:17:12.366 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T20:17:12.438 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T20:17:12.576 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T20:17:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T20:17:12.722 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T20:17:12.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T20:17:12.818 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T20:17:12.873 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T20:17:13.059 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T20:17:13.106 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T20:17:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T20:17:13.272 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T20:17:13.322 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T20:17:13.450 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T20:17:13.639 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T20:17:13.666 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T20:17:13.693 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:15.175 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T20:17:15.202 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T20:17:15.202 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:15.202 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T20:17:15.202 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:17:15.202 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T20:17:15.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:15.659 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T20:17:15.683 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:15.910 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T20:17:15.933 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T20:17:15.933 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:15.933 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T20:17:15.933 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:17:15.934 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T20:17:15.934 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:15.997 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T20:17:17.766 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T20:17:17.767 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T20:17:17.768 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T20:17:17.769 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T20:17:17.770 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:18.548 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T20:17:18.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T20:17:18.550 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:18.551 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:18.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-10T20:17:19.486 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:19.577 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:19.619 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:19.620 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:17:19.620 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:19.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-10T20:17:20.243 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:20.374 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:20.374 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 784 k 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 790 k 2024-09-10T20:17:20.375 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:20.635 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-10T20:17:20.636 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:20.636 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-10T20:17:20.636 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:20.641 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:20.641 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:20.647 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:20.648 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:20.720 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:21.138 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:21.574 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:21.815 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:21.816 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:21.816 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:21.816 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:21.816 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:21.816 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:22.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T20:17:22.582 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:54 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:22.717 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:22.717 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:22.717 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 432 k 2024-09-10T20:17:22.718 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:22.924 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 705 kB/s | 145 kB 00:00 2024-09-10T20:17:22.924 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:22.925 INFO:teuthology.orchestra.run.smithi181.stdout:Total 699 kB/s | 145 kB 00:00 2024-09-10T20:17:22.925 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:22.932 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:22.932 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:22.989 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:22.989 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:23.138 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:23.211 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:23.236 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:23.236 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:23.236 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T20:17:23.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:24.100 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:24.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-10T20:17:24.893 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:24.987 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:17:25.029 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:25.030 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:17:25.030 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:25.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T20:17:25.652 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:25.786 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:25.787 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:25.787 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T20:17:25.788 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-10T20:17:25.789 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:25.790 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 19 M 2024-09-10T20:17:25.790 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 159 M 2024-09-10T20:17:25.790 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:26.030 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 241 kB/s | 34 kB 00:00 2024-09-10T20:17:26.055 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 99 kB/s | 16 kB 00:00 2024-09-10T20:17:26.264 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-10T20:17:26.389 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-10T20:17:26.456 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-09-10T20:17:26.498 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.9 MB/s | 548 kB 00:00 2024-09-10T20:17:26.523 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-09-10T20:17:26.690 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f55 18 MB/s | 14 MB 00:00 2024-09-10T20:17:26.724 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 722 kB/s | 144 kB 00:00 2024-09-10T20:17:26.757 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 461 kB/s | 31 kB 00:00 2024-09-10T20:17:26.782 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.3 MB/s | 188 kB 00:00 2024-09-10T20:17:26.841 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.0 MB/s | 2.0 MB 00:00 2024-09-10T20:17:26.844 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:26.845 INFO:teuthology.orchestra.run.smithi181.stdout:Total 18 MB/s | 19 MB 00:01 2024-09-10T20:17:26.893 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:26.911 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:26.911 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:27.184 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:27.184 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:27.776 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:27.858 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T20:17:27.893 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T20:17:27.940 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T20:17:27.970 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T20:17:28.075 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T20:17:28.205 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T20:17:28.299 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T20:17:28.350 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T20:17:28.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T20:17:28.411 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T20:17:31.585 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T20:17:31.640 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T20:17:33.550 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T20:17:33.551 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T20:17:33.551 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T20:17:33.551 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T20:17:33.551 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T20:17:33.872 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T20:17:33.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T20:17:33.873 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T20:17:33.874 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:33.874 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:34.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T20:17:34.663 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:06 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:34.798 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T20:17:34.799 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:34.800 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-10T20:17:34.801 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-10T20:17:34.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:34.801 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-10T20:17:34.801 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-10T20:17:34.801 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:35.314 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 937 kB/s | 191 kB 00:00 2024-09-10T20:17:35.430 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-10T20:17:35.581 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-10T20:17:35.639 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 801 kB 00:00 2024-09-10T20:17:35.723 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-10T20:17:35.798 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-09-10T20:17:35.823 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 209 kB/s | 38 kB 00:00 2024-09-10T20:17:35.865 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-09-10T20:17:35.924 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-10T20:17:36.174 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 11 MB/s | 4.9 MB 00:00 2024-09-10T20:17:36.416 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-10T20:17:36.491 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.6 MB/s | 5.1 MB 00:00 2024-09-10T20:17:36.492 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:36.492 INFO:teuthology.orchestra.run.smithi181.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-10T20:17:36.844 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:36.871 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:36.871 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:37.293 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:37.293 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:38.127 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T20:17:38.173 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T20:17:38.282 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T20:17:38.316 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T20:17:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T20:17:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T20:17:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T20:17:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T20:17:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T20:17:40.610 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T20:17:41.629 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T20:17:41.655 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T20:17:44.281 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T20:17:44.281 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T20:17:44.281 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T20:17:44.281 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T20:17:44.282 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T20:17:44.724 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:44.725 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:45.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-10T20:17:45.674 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:17 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:45.809 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T20:17:45.810 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-10T20:17:45.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:45.812 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-10T20:17:45.812 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-10T20:17:45.812 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:46.156 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 209 kB/s | 26 kB 00:00 2024-09-10T20:17:46.181 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 19 kB 00:00 2024-09-10T20:17:46.224 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 260 kB/s | 49 kB 00:00 2024-09-10T20:17:46.324 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-10T20:17:46.349 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-10T20:17:46.383 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-10T20:17:46.416 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-10T20:17:46.450 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 216 kB/s | 14 kB 00:00 2024-09-10T20:17:46.475 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-10T20:17:46.509 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-10T20:17:46.543 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-10T20:17:46.567 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-10T20:17:46.601 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-10T20:17:46.603 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:46.604 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-10T20:17:46.669 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:46.684 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:46.684 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:46.774 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:46.774 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:47.315 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:47.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T20:17:47.536 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T20:17:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T20:17:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T20:17:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T20:17:47.771 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T20:17:47.873 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T20:17:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T20:17:48.652 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T20:17:48.694 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T20:17:48.727 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T20:17:48.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T20:17:48.797 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T20:17:49.596 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T20:17:49.596 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T20:17:49.596 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T20:17:49.597 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T20:17:49.928 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T20:17:49.928 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:49.928 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T20:17:49.929 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T20:17:49.930 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T20:17:49.930 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T20:17:49.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:49.930 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:50.166 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T20:17:50.737 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:22 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:50.870 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:50.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-10T20:17:50.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:50.873 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-10T20:17:50.873 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-10T20:17:50.873 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:51.286 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 769 kB/s | 154 kB 00:00 2024-09-10T20:17:51.311 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T20:17:51.370 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 988 kB/s | 58 kB 00:00 2024-09-10T20:17:51.395 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-10T20:17:51.421 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-10T20:17:51.563 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-10T20:17:51.564 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:51.565 INFO:teuthology.orchestra.run.smithi181.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-10T20:17:51.637 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:51.646 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:51.647 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:51.747 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:51.747 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:52.061 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:52.235 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T20:17:52.280 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T20:17:52.813 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T20:17:52.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T20:17:52.967 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T20:17:52.993 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T20:17:53.804 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T20:17:53.804 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T20:17:53.804 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T20:17:53.805 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T20:17:53.805 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T20:17:53.805 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T20:17:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:54.079 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:54.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-10T20:17:54.914 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:26 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:55.051 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:55.052 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-10T20:17:55.053 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:55.053 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 981 k 2024-09-10T20:17:55.053 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.9 M 2024-09-10T20:17:55.053 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:55.298 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 542 kB/s | 80 kB 00:00 2024-09-10T20:17:55.431 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.1 MB/s | 901 kB 00:00 2024-09-10T20:17:55.432 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:55.432 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.5 MB/s | 981 kB 00:00 2024-09-10T20:17:55.444 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:55.466 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:55.466 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:55.546 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:55.546 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T20:17:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T20:17:56.661 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T20:17:56.661 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:56.954 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:57.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-10T20:17:57.810 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:29 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:17:57.950 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:17:57.950 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 285 k 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-10T20:17:57.951 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:17:58.169 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-10T20:17:58.170 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:17:58.170 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-10T20:17:58.170 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:17:58.179 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:17:58.179 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:17:58.209 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:17:58.210 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:17:58.428 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:17:58.488 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:58.504 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:58.504 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:17:58.504 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T20:17:58.504 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:59.544 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T20:17:59.544 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:59.544 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:17:59.544 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T20:17:59.544 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:17:59.545 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:17:59.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-10T20:18:00.393 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:00.531 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:00.532 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 126 k 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-09-10T20:18:00.533 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:18:00.733 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 634 kB/s | 126 kB 00:00 2024-09-10T20:18:00.733 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:18:00.733 INFO:teuthology.orchestra.run.smithi181.stdout:Total 630 kB/s | 126 kB 00:00 2024-09-10T20:18:00.733 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:18:00.739 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:18:00.739 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:18:00.803 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:18:00.803 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:18:00.951 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:18:01.056 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:01.498 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:01.752 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:02.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-10T20:18:02.579 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:02.671 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:02.716 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:02.717 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:02.717 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:02.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-10T20:18:03.343 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:03.477 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:03.477 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:18:03.478 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:03.479 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-09-10T20:18:03.479 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-09-10T20:18:03.479 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:18:03.627 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 213 kB/s | 32 kB 00:00 2024-09-10T20:18:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:18:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:Total 212 kB/s | 32 kB 00:00 2024-09-10T20:18:03.628 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:18:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:18:03.632 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:18:03.654 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:18:03.655 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:18:03.733 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:18:03.850 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:04.230 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:04.467 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:18:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:04.467 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:04.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-10T20:18:05.187 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:05.276 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:05.320 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:05.321 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:05.321 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:05.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-10T20:18:05.947 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:06.036 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:06.079 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:06.080 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:06.080 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:06.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-10T20:18:06.705 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:06.795 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:06.838 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:06.839 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:06.840 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:06.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-10T20:18:07.465 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:07.554 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:07.598 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:07.598 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:07.599 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:07.668 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-10T20:18:08.232 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:08.323 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:08.365 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:08.366 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:08.366 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:08.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-10T20:18:08.998 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:09.088 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T20:18:09.132 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:09.133 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-10T20:18:09.133 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:09.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-10T20:18:09.765 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:09.899 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:09.899 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:09.899 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-09-10T20:18:09.900 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:18:10.089 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-10T20:18:10.089 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:18:10.089 INFO:teuthology.orchestra.run.smithi181.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-10T20:18:10.090 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:18:10.096 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:18:10.096 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:18:10.160 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:18:10.161 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:18:10.304 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:18:10.394 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:10.771 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:11.050 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:11.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-10T20:18:11.890 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:12.034 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:18:12.035 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:12.036 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:18:12.036 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:12.036 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.2 M 2024-09-10T20:18:12.036 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 14 M 2024-09-10T20:18:12.036 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:18:12.368 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-10T20:18:12.368 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:18:12.369 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-10T20:18:12.369 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:18:12.379 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:18:12.469 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:18:12.469 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:18:12.928 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:18:13.002 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:13.028 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:13.028 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T20:18:13.029 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T20:18:13.029 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:18:13.029 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T20:18:13.029 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:14.133 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:14.134 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:18:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:14.134 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:14.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-10T20:18:14.932 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 08:16:28 PM UTC. 2024-09-10T20:18:15.067 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-10T20:18:15.067 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-10T20:18:15.067 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:15.068 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-09-10T20:18:15.069 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 490 k 2024-09-10T20:18:15.069 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-10T20:18:15.288 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 778 kB/s | 170 kB 00:00 2024-09-10T20:18:15.289 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-10T20:18:15.289 INFO:teuthology.orchestra.run.smithi181.stdout:Total 772 kB/s | 170 kB 00:00 2024-09-10T20:18:15.289 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-10T20:18:15.296 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-10T20:18:15.297 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-10T20:18:15.361 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-10T20:18:15.362 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-10T20:18:15.513 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-10T20:18:15.610 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:16.012 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:16.259 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T20:18:16.260 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:16.260 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-10T20:18:16.260 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T20:18:16.260 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-10T20:18:16.260 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-10T20:18:16.495 DEBUG:teuthology.parallel:result is None 2024-09-10T20:18:16.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:18:16.646 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T20:18:16.670 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T20:18:16.670 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T20:18:16.670 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T20:18:16.672 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T20:18:16.673 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:18:16.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:18:16.737 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T20:18:16.738 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:18:16.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:18:16.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:18:16.869 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T20:18:16.869 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:18:16.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:18:16.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:18:17.000 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T20:18:17.000 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:18:17.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:18:17.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:18:17.131 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T20:18:17.150 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:18:17.150 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T20:18:17.150 INFO:tasks.workunit:timeout=3h 2024-09-10T20:18:17.150 INFO:tasks.workunit:cleanup=True 2024-09-10T20:18:17.151 DEBUG:teuthology.orchestra.run.smithi181:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:18:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:18:17.189 INFO:teuthology.orchestra.run.smithi181.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T20:18:17.190 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:18:17.244 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T20:18:17.244 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T20:18:17.299 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:18:17.355 INFO:tasks.workunit.client.0.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T20:19:06.808 INFO:tasks.workunit.client.0.smithi181.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.809 INFO:tasks.workunit.client.0.smithi181.stderr: git switch -c 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr:Or undo this operation with: 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr: git switch - 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr: 2024-09-10T20:19:06.810 INFO:tasks.workunit.client.0.smithi181.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:19:06.812 DEBUG:teuthology.orchestra.run.smithi181:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T20:19:06.868 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-10T20:19:06.869 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T20:19:06.923 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-09-10T20:19:06.924 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-09-10T20:19:06.924 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-09-10T20:19:06.995 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:19:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:19:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:19:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:19:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:19:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-mapper 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-09-10T20:19:07.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-09-10T20:19:07.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none ' 2024-09-10T20:19:07.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-10T20:19:07.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-10T20:19:07.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-09-10T20:19:07.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:19:07.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:19:07.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:19:07.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:19:07.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:19:07.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:19:07.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:19:07.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:19:07.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:19:07.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:19:07.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:19:07.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:19:07.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:19:07.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:19:07.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:19:07.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:19:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:19:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:19:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-10T20:19:07.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:19:07.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:07.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:19:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27299 2024-09-10T20:19:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:19:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:19:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2024-09-10T20:19:07.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:19:07.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:07.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:19:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27299 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-09-10T20:19:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-09-10T20:19:07.022 INFO:tasks.workunit.client.0.smithi181.stdout:Dir: td/osd-mapper 2024-09-10T20:19:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-09-10T20:19:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2024-09-10T20:19:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=test 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=test 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local extra_pars= 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-10T20:19:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-10T20:19:08.754 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 21b291ff-99f0-450a-915e-0d31a3d0b488 2024-09-10T20:19:08.966 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:19:09.046 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:09.042+0000 7f484f217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:09.044+0000 7f484f217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:09.045+0000 7f484f217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:09.046+0000 7f484f217640 -1 bdev(0x558daf7afc00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:19:09.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:09.046+0000 7f484f217640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-09-10T20:19:15.719 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:19:16.070 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:19:16.102 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:16.099+0000 7f5d43ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:16.104 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:16.101+0000 7f5d43ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:16.106 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:16.103+0000 7f5d43ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:16.352 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:19:17.629 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:19:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:17.952+0000 7f5d43ae2640 -1 Falling back to public interface 2024-09-10T20:19:18.912 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:19:19.612 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:19.609+0000 7f5d43ae2640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:19:20.200 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:19:21.509 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:19:22.795 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:19:24.083 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:19:25.373 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:19:26.660 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:19:27.948 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:19:29.337 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:19:30.623 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:19:32.152 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:19:33.767 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:19:35.054 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:19:36.375 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:19:37.809 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:19:39.280 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:19:41.098 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:19:43.394 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:19:45.078 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T20:19:45.367 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/880626726,v1:127.0.0.1:6803/880626726] [v2:127.0.0.1:6804/880626726,v1:127.0.0.1:6805/880626726] exists,up 21b291ff-99f0-450a-915e-0d31a3d0b488 2024-09-10T20:19:45.370 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 37f792b8-2497-4d86-bd57-ef4140e0aed6 2024-09-10T20:19:45.747 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:19:45.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:45.788+0000 7f530ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:45.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:45.827+0000 7f530ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:45.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:45.828+0000 7f530ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:45.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:45.828+0000 7f530ae0d640 -1 bdev(0x55f348c90800 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:19:45.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:45.828+0000 7f530ae0d640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-09-10T20:19:52.229 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:19:52.585 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:19:52.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:52.614+0000 7ff644cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:52.620 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:52.617+0000 7ff644cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:52.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:52.618+0000 7ff644cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:52.883 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:19:54.157 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:19:54.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:54.222+0000 7ff644cf1640 -1 Falling back to public interface 2024-09-10T20:19:55.445 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:19:55.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:19:55.834+0000 7ff644cf1640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:19:56.723 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:19:58.009 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:19:59.294 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:20:00.656 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:20:01.960 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:20:03.241 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:20:04.527 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:20:05.816 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:20:07.101 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:20:08.386 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:20:09.840 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:20:11.259 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:20:12.836 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:20:14.359 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:20:16.217 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:20:17.686 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:20:19.073 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:20:19.353 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4028619580,v1:127.0.0.1:6811/4028619580] [v2:127.0.0.1:6812/4028619580,v1:127.0.0.1:6813/4028619580] exists,up 37f792b8-2497-4d86-bd57-ef4140e0aed6 2024-09-10T20:20:19.357 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 853f4461-6584-4409-9ffb-204e015aa7d9 2024-09-10T20:20:19.700 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:20:19.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:19.744+0000 7f325b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:19.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:19.781+0000 7f325b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:19.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:19.782+0000 7f325b611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:19.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:19.783+0000 7f325b611640 -1 bdev(0x55655ceb8800 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:20:19.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:19.783+0000 7f325b611640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-09-10T20:20:26.041 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:20:26.399 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:20:26.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:26.427+0000 7fbcf3e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:26.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:26.429+0000 7fbcf3e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:26.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:26.430+0000 7fbcf3e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:26.673 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:20:27.970 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:20:28.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:28.041+0000 7fbcf3e14640 -1 Falling back to public interface 2024-09-10T20:20:29.255 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:20:29.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:20:29.687+0000 7fbcf3e14640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:20:30.541 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:20:31.942 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:20:33.224 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:20:34.540 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:20:35.903 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:20:37.192 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:20:38.521 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:20:39.835 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:20:41.120 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:20:42.439 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:20:43.924 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:20:45.515 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:20:46.844 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:20:48.515 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:20:50.137 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:20:51.685 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:20:53.455 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:20:53.752 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4164213081,v1:127.0.0.1:6819/4164213081] [v2:127.0.0.1:6820/4164213081,v1:127.0.0.1:6821/4164213081] exists,up 853f4461-6584-4409-9ffb-204e015aa7d9 2024-09-10T20:20:54.140 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T20:20:56.198 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836500 2024-09-10T20:20:56.560 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672973 2024-09-10T20:20:56.951 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509447 2024-09-10T20:21:02.723 INFO:tasks.workunit.client.0.smithi181.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-10T20:21:02.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-09-10T20:21:02.817 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T20:21:02.817 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.817 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.830 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T20:21:02.830 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.830 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.843 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T20:21:02.843 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.843 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-09-10T20:21:02.957 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T20:21:02.957 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.957 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.971 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T20:21:02.971 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.971 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.983 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T20:21:02.984 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-10T20:21:02.984 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:02.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-09-10T20:21:02.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-09-10T20:21:02.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-09-10T20:21:02.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-09-10T20:21:02.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-09-10T20:21:02.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-10T20:21:02.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-10T20:21:03.311 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap01 2024-09-10T20:21:03.520 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap02 2024-09-10T20:21:03.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-09-10T20:21:03.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-10T20:21:03.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-10T20:21:03.973 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap13 2024-09-10T20:21:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-09-10T20:21:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-10T20:21:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-10T20:21:04.458 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap24 2024-09-10T20:21:04.701 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap25 2024-09-10T20:21:04.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 12778 2024-09-10T20:21:04.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-09-10T20:21:05.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-10T20:21:05.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-09-10T20:21:05.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-09-10T20:21:05.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-10T20:21:05.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-10T20:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-09-10T20:21:05.580 INFO:tasks.workunit.client.0.smithi181.stdout:pgid is 1.3 (primary: osd.1) 2024-09-10T20:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-09-10T20:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-09-10T20:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: set_query_debug: local pgid=1.3 2024-09-10T20:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: ceph pg dump pgs_brief 2024-09-10T20:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-10T20:21:05.838 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs_brief 2024-09-10T20:21:05.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: local prim_osd=1 2024-09-10T20:21:05.850 INFO:tasks.workunit.client.0.smithi181.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:310: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: get_asok_path osd.1 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27299/ceph-osd.1.asok 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: CEPH_ARGS= 2024-09-10T20:21:05.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.27299/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-09-10T20:21:05.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:05.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-10T20:21:05.969 INFO:tasks.workunit.client.0.smithi181.stdout:{"success":true}{ 2024-09-10T20:21:05.969 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "objxxx", 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 1, 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap01" 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap02" 2024-09-10T20:21:05.970 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 5, 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap13" 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:05.971 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 6, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 4, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap24" 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap25" 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:05.972 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 5, 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: "id": "head", 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [], 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 6 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T20:21:05.973 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:05.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-10T20:21:06.063 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:06.063 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:21:06.063 INFO:tasks.workunit.client.0.smithi181.stdout: "must": true, 2024-09-10T20:21:06.063 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "0.000000" 2024-09-10T20:21:06.063 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:06.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-09-10T20:21:09.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-10T20:21:09.343 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:09.344 INFO:tasks.workunit.client.0.smithi181.stdout:1.3 1 0 0 0 0 5 0 0 1 0 1 active+clean 2024-09-10T20:20:55.428997+0000 24'1 24:24 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.344 INFO:tasks.workunit.client.0.smithi181.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:56.066621+0000 0'0 29:35 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.344 INFO:tasks.workunit.client.0.smithi181.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:55.278643+0000 0'0 29:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.344 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:55.428991+0000 0'0 24:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.344 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:09.345 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:09.345 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:21:09.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-09-10T20:21:09.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-10T20:21:09.614 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout:1.3 1 0 0 0 0 5 0 0 1 0 1 active+clean 2024-09-10T20:20:55.428997+0000 24'1 24:24 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:56.066621+0000 0'0 29:35 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:55.278643+0000 0'0 29:39 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:20:55.428991+0000 0'0 24:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:09.615 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:21:09.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-09-10T20:21:10.090 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T20:21:10.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-09-10T20:21:10.510 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T20:21:10.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-09-10T20:21:15.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-09-10T20:21:15.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-10T20:21:15.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:21:15.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:21:15.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:21:15.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:21:15.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:21:15.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:21:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-10T20:21:15.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:15.857+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:15.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:15.857+0000 7f96abc68980 0 bdev(0x55ac4a013c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:15.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:15.857+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:15.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:15.894+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) close 2024-09-10T20:21:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.119+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.119+0000 7f96abc68980 0 bdev(0x55ac4a013c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.119+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 0 bdev(0x55ac4a18e400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bluefs mount 2024-09-10T20:21:16.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.120+0000 7f96abc68980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:16.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.156+0000 7f96abc68980 1 bluefs mount shared_bdev_used = 0 2024-09-10T20:21:16.159 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.156+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:16.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.193+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:16.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.193+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-10T20:21:16.197 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.193+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.193+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.193+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.194+0000 7f96abc68980 1 freelist init 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.194+0000 7f96abc68980 1 freelist _read_cfg 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.194+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.194+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 27 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffba000, fragmentation 9.91824e-07 2024-09-10T20:21:16.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.195+0000 7f96abc68980 1 bluefs umount 2024-09-10T20:21:16.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.195+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) close 2024-09-10T20:21:16.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.382+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:16.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.382+0000 7f96abc68980 0 bdev(0x55ac4a18e400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:16.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.382+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:16.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.382+0000 7f96abc68980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-10T20:21:16.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.382+0000 7f96abc68980 1 bluefs mount 2024-09-10T20:21:16.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.395+0000 7f96abc68980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:16.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.408+0000 7f96abc68980 1 bluefs mount shared_bdev_used = 27459584 2024-09-10T20:21:16.412 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.408+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:16.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.439+0000 7f96abc68980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 0b 05 58 02 00 00 52 a2 b3 0e 46 b6 4d 87 be 7e |..X...R...F.M..~| 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 50 fa ed 7d fd 20 00 00 00 00 1f 00 00 00 00 00 |P..}. ..........| 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:* 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-10T20:21:16.445 INFO:tasks.workunit.client.0.smithi181.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-10T20:21:16.446 INFO:tasks.workunit.client.0.smithi181.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-10T20:21:16.447 INFO:tasks.workunit.client.0.smithi181.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000220 72 65 1f 00 00 00 05 00 00 00 21 b2 91 ff 99 f0 |re........!.....| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000230 45 0a 91 5e 0d 31 a3 d0 b4 88 00 00 00 00 00 00 |E..^.1..........| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000240 00 00 1f 00 00 00 e0 a9 e0 66 25 fb 3d 0a 01 00 |.........f%.=...| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1f 00 00 00 |..............| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:0000025e 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:16.448 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:0000003f 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:16.449 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.450 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:16.451 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:16.452 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.440+0000 7f96abc68980 1 bluefs umount 2024-09-10T20:21:16.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.440+0000 7f96abc68980 1 bdev(0x55ac4a18e400 td/osd-mapper/0/block) close 2024-09-10T20:21:16.647 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.644+0000 7f96abc68980 1 freelist shutdown 2024-09-10T20:21:16.648 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.644+0000 7f96abc68980 1 bdev(0x55ac4a013c00 td/osd-mapper/0/block) close 2024-09-10T20:21:16.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:16.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-10T20:21:16.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-09-10T20:21:16.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.911+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:16.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.911+0000 7fa5225d6980 0 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:16.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.912+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:16.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:16.920+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) close 2024-09-10T20:21:17.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.177+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:17.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.177+0000 7fa5225d6980 0 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:17.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.177+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 0 bdev(0x55e0ca960400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.178+0000 7fa5225d6980 1 bluefs mount 2024-09-10T20:21:17.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.179+0000 7fa5225d6980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:17.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.192+0000 7fa5225d6980 1 bluefs mount shared_bdev_used = 0 2024-09-10T20:21:17.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.192+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.230+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.230+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.230+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.230+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.230+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-10T20:21:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 freelist init 2024-09-10T20:21:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 freelist _read_cfg 2024-09-10T20:21:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-10T20:21:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 18 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffba000, fragmentation 6.485e-07 2024-09-10T20:21:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 bluefs umount 2024-09-10T20:21:17.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.231+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) close 2024-09-10T20:21:17.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:17.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 0 bdev(0x55e0ca960400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:17.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:17.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-10T20:21:17.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 1 bluefs mount 2024-09-10T20:21:17.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.444+0000 7fa5225d6980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:17.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.451+0000 7fa5225d6980 1 bluefs mount shared_bdev_used = 27459584 2024-09-10T20:21:17.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.451+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:17.486 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.482+0000 7fa5225d6980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.483+0000 7fa5225d6980 1 bluefs umount 2024-09-10T20:21:17.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.483+0000 7fa5225d6980 1 bdev(0x55e0ca960400 td/osd-mapper/2/block) close 2024-09-10T20:21:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.706+0000 7fa5225d6980 1 freelist shutdown 2024-09-10T20:21:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:17.706+0000 7fa5225d6980 1 bdev(0x55e0ca7e5c00 td/osd-mapper/2/block) close 2024-09-10T20:21:17.947 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:17.947 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:17.947 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:17.947 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:17.948 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:17.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:17.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-09-10T20:21:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-09-10T20:21:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-10T20:21:18.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-10T20:21:18.546 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 0b 05 58 02 00 00 52 a2 b3 0e 46 b6 4d 87 be 7e |..X...R...F.M..~| 2024-09-10T20:21:18.546 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 50 fa ed 7d fd 20 02 00 00 00 1f 00 00 00 00 00 |P..}. ..........| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:* 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-10T20:21:18.547 INFO:tasks.workunit.client.0.smithi181.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-10T20:21:18.548 INFO:tasks.workunit.client.0.smithi181.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-10T20:21:18.549 INFO:tasks.workunit.client.0.smithi181.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-10T20:21:18.550 INFO:tasks.workunit.client.0.smithi181.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-10T20:21:18.551 INFO:tasks.workunit.client.0.smithi181.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout:00000220 72 65 1f 00 00 00 0f 00 00 00 85 3f 44 61 65 84 |re.........?Dae.| 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout:00000230 44 09 9f fb 20 4e 01 5a a7 d9 00 00 00 00 00 00 |D... N.Z........| 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout:00000240 00 00 1f 00 00 00 24 aa e0 66 f9 22 ba 20 01 00 |......$..f.". ..| 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1f 00 00 00 |..............| 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout:0000025e 2024-09-10T20:21:18.552 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:18.553 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:18.554 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:0000003f 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:18.555 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:18.556 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:18.557 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:18.558 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:18.559 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:18.560 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:18.560 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.560 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:18.560 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:18.560 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:18.561 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:18.561 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:18.561 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:18.561 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:18.562 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:18.563 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.563 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-10T20:21:18.563 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:18.563 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-10T20:21:18.563 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:19.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-10T20:21:19.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-10T20:21:19.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:21:19.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-09-10T20:21:20.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-09-10T20:21:20.057 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-09-10T20:21:20.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-09-10T20:21:20.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:20.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-09-10T20:21:20.060 INFO:tasks.workunit.client.0.smithi181.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-09-10T20:21:20.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-09-10T20:21:20.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-10T20:21:20.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-10T20:21:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.083+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.083+0000 7efdca120980 0 bdev(0x5598fdb11c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:20.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.083+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:20.087 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.084+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) close 2024-09-10T20:21:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.345+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:20.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.345+0000 7efdca120980 0 bdev(0x5598fdb11c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.345+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-10T20:21:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 0 bdev(0x5598fdc8c400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-10T20:21:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bluefs mount 2024-09-10T20:21:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.346+0000 7efdca120980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:20.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.353+0000 7efdca120980 1 bluefs mount shared_bdev_used = 0 2024-09-10T20:21:20.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.353+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-10T20:21:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-10T20:21:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-10T20:21:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-10T20:21:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 freelist init 2024-09-10T20:21:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 freelist _read_cfg 2024-09-10T20:21:20.396 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.391+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-10T20:21:20.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.392+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 27 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffba000, fragmentation 9.91824e-07 2024-09-10T20:21:20.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.392+0000 7efdca120980 1 bluefs umount 2024-09-10T20:21:20.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.392+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) close 2024-09-10T20:21:20.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-10T20:21:20.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 0 bdev(0x5598fdc8c400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-10T20:21:20.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:20.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-10T20:21:20.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 1 bluefs mount 2024-09-10T20:21:20.616 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.612+0000 7efdca120980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:20.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.618+0000 7efdca120980 1 bluefs mount shared_bdev_used = 27459584 2024-09-10T20:21:20.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.618+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:20.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.649+0000 7efdca120980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 0b 05 58 02 00 00 52 a2 b3 0e 46 b6 4d 87 be 7e |..X...R...F.M..~| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 50 fa ed 7d fd 20 00 00 00 00 1f 00 00 00 00 00 |P..}. ..........| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:* 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-10T20:21:20.655 INFO:tasks.workunit.client.0.smithi181.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-10T20:21:20.656 INFO:tasks.workunit.client.0.smithi181.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000220 72 65 1f 00 00 00 05 00 00 00 21 b2 91 ff 99 f0 |re........!.....| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000230 45 0a 91 5e 0d 31 a3 d0 b4 88 00 00 00 00 00 00 |E..^.1..........| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000240 00 00 1f 00 00 00 e0 a9 e0 66 25 fb 3d 0a 01 00 |.........f%.=...| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1f 00 00 00 |..............| 2024-09-10T20:21:20.657 INFO:tasks.workunit.client.0.smithi181.stdout:0000025e 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-10T20:21:20.658 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:0000003f 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:20.659 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:20.660 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:20.661 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:20.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.650+0000 7efdca120980 1 bluefs umount 2024-09-10T20:21:20.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.650+0000 7efdca120980 1 bdev(0x5598fdc8c400 td/osd-mapper/0/block) close 2024-09-10T20:21:20.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.875+0000 7efdca120980 1 freelist shutdown 2024-09-10T20:21:20.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:20.875+0000 7efdca120980 1 bdev(0x5598fdb11c00 td/osd-mapper/0/block) close 2024-09-10T20:21:21.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-10T20:21:21.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-10T20:21:21.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:21:22.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:22.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:22.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-10T20:21:22.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-10T20:21:22.177 INFO:tasks.workunit.client.0.smithi181.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:22.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-10T20:21:22.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.DgANrJiIAW_the_val 2024-09-10T20:21:22.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:22.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.DgANrJiIAW_the_val' 2024-09-10T20:21:22.180 INFO:tasks.workunit.client.0.smithi181.stdout:Value dumped in: /tmp/tmp.DgANrJiIAW_the_val 2024-09-10T20:21:22.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.DgANrJiIAW_the_val 2024-09-10T20:21:22.774 INFO:tasks.workunit.client.0.smithi181.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-10T20:21:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.DgANrJiIAW_the_val 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-10T20:21:23.251 INFO:tasks.workunit.client.0.smithi181.stdout:0000073 2024-09-10T20:21:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-10T20:21:23.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:25.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.DgANrJiIAW_the_val 2024-09-10T20:21:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-10T20:21:26.365 INFO:tasks.workunit.client.0.smithi181.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-09-10T20:21:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-09-10T20:21:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-10T20:21:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-10T20:21:26.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.387+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-10T20:21:26.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.387+0000 7f6ade5da980 0 bdev(0x55ef93409c00 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-10T20:21:26.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.387+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:26.415 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.410+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) close 2024-09-10T20:21:26.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.653+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-10T20:21:26.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.653+0000 7f6ade5da980 0 bdev(0x55ef93409c00 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-10T20:21:26.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.653+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:26.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 0 bdev(0x55ef93584400 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bluefs mount 2024-09-10T20:21:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.661+0000 7f6ade5da980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:26.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.669+0000 7f6ade5da980 1 bluefs mount shared_bdev_used = 0 2024-09-10T20:21:26.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.669+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:26.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.706+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:26.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.706+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2051 2024-09-10T20:21:26.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.706+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-09-10T20:21:26.711 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.706+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.706+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 freelist init 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 freelist _read_cfg 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 19 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffba000, fragmentation 6.86647e-07 2024-09-10T20:21:26.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 bluefs umount 2024-09-10T20:21:26.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.707+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) close 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 0 bdev(0x55ef93584400 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 1 bluefs mount 2024-09-10T20:21:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.923+0000 7f6ade5da980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:26.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.930+0000 7f6ade5da980 1 bluefs mount shared_bdev_used = 27459584 2024-09-10T20:21:26.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.930+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:26.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.961+0000 7f6ade5da980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:26.968 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-10T20:21:26.968 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 0b 05 58 02 00 00 52 a2 b3 0e 46 b6 4d 87 be 7e |..X...R...F.M..~| 2024-09-10T20:21:26.968 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 50 fa ed 7d fd 20 01 00 00 00 1f 00 00 00 00 00 |P..}. ..........| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:* 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-10T20:21:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-10T20:21:26.970 INFO:tasks.workunit.client.0.smithi181.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-10T20:21:26.971 INFO:tasks.workunit.client.0.smithi181.stdout:00000220 72 65 1f 00 00 00 0a 00 00 00 37 f7 92 b8 24 97 |re........7...$.| 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:00000230 4d 86 bd 57 ef 41 40 e0 ae d6 00 00 00 00 00 00 |M..W.A@.........| 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:00000240 00 00 1f 00 00 00 01 aa e0 66 91 6f 0d 32 01 00 |.........f.o.2..| 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1f 00 00 00 |..............| 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:0000025e 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:26.972 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:26.973 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:0000003f 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.974 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:26.975 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:26.976 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:26.977 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.962+0000 7f6ade5da980 1 bluefs umount 2024-09-10T20:21:26.978 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:26.962+0000 7f6ade5da980 1 bdev(0x55ef93584400 td/osd-mapper/1/block) close 2024-09-10T20:21:27.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:27.185+0000 7f6ade5da980 1 freelist shutdown 2024-09-10T20:21:27.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:27.185+0000 7f6ade5da980 1 bdev(0x55ef93409c00 td/osd-mapper/1/block) close 2024-09-10T20:21:27.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-10T20:21:27.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-10T20:21:27.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:21:28.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:28.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:28.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-10T20:21:28.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-10T20:21:28.486 INFO:tasks.workunit.client.0.smithi181.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:28.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-10T20:21:28.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.iOHw7rdLu6_the_val 2024-09-10T20:21:28.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:28.488 INFO:tasks.workunit.client.0.smithi181.stdout:Value dumped in: /tmp/tmp.iOHw7rdLu6_the_val 2024-09-10T20:21:28.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.iOHw7rdLu6_the_val' 2024-09-10T20:21:28.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.iOHw7rdLu6_the_val 2024-09-10T20:21:29.087 INFO:tasks.workunit.client.0.smithi181.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-10T20:21:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:29.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.iOHw7rdLu6_the_val 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-10T20:21:29.547 INFO:tasks.workunit.client.0.smithi181.stdout:0000073 2024-09-10T20:21:29.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-10T20:21:29.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:31.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.iOHw7rdLu6_the_val 2024-09-10T20:21:32.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:32.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:32.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-09-10T20:21:32.668 INFO:tasks.workunit.client.0.smithi181.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-09-10T20:21:32.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-09-10T20:21:32.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-10T20:21:32.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-10T20:21:32.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.688+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:32.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.688+0000 7f760bb79980 0 bdev(0x5559ab819c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:32.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.688+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:32.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.701+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) close 2024-09-10T20:21:32.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.959+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:32.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.959+0000 7f760bb79980 0 bdev(0x5559ab819c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:32.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.959+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 0 bdev(0x5559ab994400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bluefs mount 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.960+0000 7f760bb79980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:32.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.981+0000 7f760bb79980 1 bluefs mount shared_bdev_used = 0 2024-09-10T20:21:32.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:32.981+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:33.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.018+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:33.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.018+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-10T20:21:33.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.018+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-10T20:21:33.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.018+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.018+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.019+0000 7f760bb79980 1 freelist init 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.019+0000 7f760bb79980 1 freelist _read_cfg 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.019+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.019+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 18 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffba000, fragmentation 6.485e-07 2024-09-10T20:21:33.024 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.019+0000 7f760bb79980 1 bluefs umount 2024-09-10T20:21:33.025 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.020+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) close 2024-09-10T20:21:33.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.222+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-10T20:21:33.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.222+0000 7f760bb79980 0 bdev(0x5559ab994400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-10T20:21:33.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.222+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-10T20:21:33.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.222+0000 7f760bb79980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-10T20:21:33.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.222+0000 7f760bb79980 1 bluefs mount 2024-09-10T20:21:33.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.223+0000 7f760bb79980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-10T20:21:33.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.229+0000 7f760bb79980 1 bluefs mount shared_bdev_used = 27459584 2024-09-10T20:21:33.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.229+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-10T20:21:33.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.261+0000 7f760bb79980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-10T20:21:33.267 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 0b 05 58 02 00 00 52 a2 b3 0e 46 b6 4d 87 be 7e |..X...R...F.M..~| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 50 fa ed 7d fd 20 02 00 00 00 1f 00 00 00 00 00 |P..}. ..........| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:* 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-10T20:21:33.268 INFO:tasks.workunit.client.0.smithi181.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-10T20:21:33.269 INFO:tasks.workunit.client.0.smithi181.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000220 72 65 1f 00 00 00 0f 00 00 00 85 3f 44 61 65 84 |re.........?Dae.| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000230 44 09 9f fb 20 4e 01 5a a7 d9 00 00 00 00 00 00 |D... N.Z........| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000240 00 00 1f 00 00 00 24 aa e0 66 f9 22 ba 20 01 00 |......$..f.". ..| 2024-09-10T20:21:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1f 00 00 00 |..............| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:0000025e 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:33.271 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:0000003f 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout:00000047 2024-09-10T20:21:33.272 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:33.273 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.274 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:0000003b 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-10T20:21:33.275 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:33.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.262+0000 7f760bb79980 1 bluefs umount 2024-09-10T20:21:33.276 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.262+0000 7f760bb79980 1 bdev(0x5559ab994400 td/osd-mapper/2/block) close 2024-09-10T20:21:33.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.484+0000 7f760bb79980 1 freelist shutdown 2024-09-10T20:21:33.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:33.484+0000 7f760bb79980 1 bdev(0x5559ab819c00 td/osd-mapper/2/block) close 2024-09-10T20:21:33.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-10T20:21:33.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-10T20:21:33.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:21:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-10T20:21:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-10T20:21:34.789 INFO:tasks.workunit.client.0.smithi181.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:34.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-10T20:21:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.qduZWtLGJD_the_val 2024-09-10T20:21:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:34.791 INFO:tasks.workunit.client.0.smithi181.stdout:Value dumped in: /tmp/tmp.qduZWtLGJD_the_val 2024-09-10T20:21:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.qduZWtLGJD_the_val' 2024-09-10T20:21:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.qduZWtLGJD_the_val 2024-09-10T20:21:35.382 INFO:tasks.workunit.client.0.smithi181.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-10T20:21:35.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:35.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.qduZWtLGJD_the_val 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-10T20:21:35.852 INFO:tasks.workunit.client.0.smithi181.stdout:0000073 2024-09-10T20:21:35.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-10T20:21:35.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:21:37.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.qduZWtLGJD_the_val 2024-09-10T20:21:38.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:38.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 ' 2024-09-10T20:21:38.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 2024-09-10T20:21:38.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:38.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:21:38.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/' 2024-09-10T20:21:38.974 INFO:tasks.workunit.client.0.smithi181.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ 2024-09-10T20:21:38.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-09-10T20:21:38.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-09-10T20:21:38.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:38.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:38.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:21:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-10T20:21:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2024-09-10T20:21:38.981 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:21:38.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:21:38.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:21:39.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:39.024+0000 7f4f52fb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:39.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:39.027+0000 7f4f52fb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:39.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:39.028+0000 7f4f52fb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:39.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:21:39.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:39.285 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:21:39.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:39.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:21:39.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:40.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:40.390+0000 7f4f52fb9640 -1 Falling back to public interface 2024-09-10T20:21:40.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:40.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:40.613 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:21:40.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:21:40.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:40.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:21:40.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:41.842 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:41.838+0000 7f4f52fb9640 -1 osd.0 31 log_to_monitors true 2024-09-10T20:21:41.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:41.930 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:21:41.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:41.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:21:41.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:41.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:21:42.324 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 34 up_thru 17 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/3572648719,v1:127.0.0.1:6803/3572648719] [v2:127.0.0.1:6804/3572648719,v1:127.0.0.1:6805/3572648719] exists,up 21b291ff-99f0-450a-915e-0d31a3d0b488 2024-09-10T20:21:42.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:21:42.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-09-10T20:21:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:21:42.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:42.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:21:42.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:21:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:21:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2024-09-10T20:21:42.335 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:21:42.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:21:42.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:21:42.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:42.368+0000 7f6fd1816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:42.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:42.370+0000 7f6fd1816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:42.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:42.371+0000 7f6fd1816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:42.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:42.657 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:21:42.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:42.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:21:42.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:43.251+0000 7f6fd1816640 -1 Falling back to public interface 2024-09-10T20:21:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:43.953 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:21:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:21:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:21:44.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:44.643 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:44.639+0000 7f6fd1816640 -1 osd.1 31 log_to_monitors true 2024-09-10T20:21:45.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:45.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:45.260 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:21:45.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:21:45.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:45.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:21:45.623 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/2753252042,v1:127.0.0.1:6811/2753252042] [v2:127.0.0.1:6812/2753252042,v1:127.0.0.1:6813/2753252042] exists,up 37f792b8-2497-4d86-bd57-ef4140e0aed6 2024-09-10T20:21:45.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:45.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:45.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:45.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:45.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2024-09-10T20:21:45.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:21:45.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:21:45.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:21:45.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:21:45.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:21:45.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:21:45.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2024-09-10T20:21:45.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:21:45.635 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=52a2b30e-46b6-4d87-be7e-50faed7dfd20 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27299/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:21:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:21:45.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:45.668+0000 7fcd626e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:45.678+0000 7fcd626e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:45.679+0000 7fcd626e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:45.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:45.925 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:21:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:21:46.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:46.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:46.800+0000 7fcd626e4640 -1 Falling back to public interface 2024-09-10T20:21:47.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:47.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:47.257 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:21:47.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:21:47.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:47.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:21:47.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:21:48.285 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:21:48.278+0000 7fcd626e4640 -1 osd.2 31 log_to_monitors true 2024-09-10T20:21:48.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:21:48.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:48.561 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:21:48.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:21:48.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:48.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:21:48.912 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 41 up_thru 17 down_at 32 last_clean_interval [15,31) [v2:127.0.0.1:6818/570695108,v1:127.0.0.1:6819/570695108] [v2:127.0.0.1:6820/570695108,v1:127.0.0.1:6821/570695108] exists,up 853f4461-6584-4409-9ffb-204e015aa7d9 2024-09-10T20:21:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-09-10T20:21:49.914 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-09-10T20:21:49.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-09-10T20:21:49.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:49.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-09-10T20:21:50.025 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:50.025 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:50.025 INFO:tasks.workunit.client.0.smithi181.stdout: "release": "squid", 2024-09-10T20:21:50.025 INFO:tasks.workunit.client.0.smithi181.stdout: "release_type": "dev" 2024-09-10T20:21:50.025 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:50.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:50.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-09-10T20:21:50.129 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:50.130 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:50.130 INFO:tasks.workunit.client.0.smithi181.stdout: "release": "squid", 2024-09-10T20:21:50.130 INFO:tasks.workunit.client.0.smithi181.stdout: "release_type": "dev" 2024-09-10T20:21:50.130 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-10T20:21:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-09-10T20:21:50.233 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:50.233 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:50.233 INFO:tasks.workunit.client.0.smithi181.stdout: "release": "squid", 2024-09-10T20:21:50.233 INFO:tasks.workunit.client.0.smithi181.stdout: "release_type": "dev" 2024-09-10T20:21:50.233 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:50.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "objxxx", 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 1, 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap01" 2024-09-10T20:21:50.271 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 2, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap02" 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 5, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 3, 2024-09-10T20:21:50.272 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap13" 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 6, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [ 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 4, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap24" 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "id": 5, 2024-09-10T20:21:50.273 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "snap25" 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 5, 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: "overlaps": [] 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: "id": "head", 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: "snapshots": [], 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 6 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T20:21:50.274 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:50.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-09-10T20:21:50.657 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is unset 2024-09-10T20:21:50.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-09-10T20:21:51.067 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is unset 2024-09-10T20:21:51.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-10T20:21:51.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-10T20:21:51.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-09-10T20:21:51.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-10T20:21:51.634 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-10T20:21:49.845044+0000 29'7 43:69 [1,2,0] 1 [1,2,0] 1 29'7 2024-09-10T20:21:06.923330+0000 29'7 2024-09-10T20:21:06.923330+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:21:06.923330+0000 4 0 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-10T20:21:46.030803+0000 0'0 39:33 [0,1] 0 [0,1] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-10T20:21:46.030798+0000 0'0 39:27 [0,1] 0 [0,1] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.928679+0000 0'0 43:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:51.635 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:21:51.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-09-10T20:21:53.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-10T20:21:53.749 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:53.750 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:21:53.750 INFO:tasks.workunit.client.0.smithi181.stdout: "must": true, 2024-09-10T20:21:53.750 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "0.000000" 2024-09-10T20:21:53.750 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:53.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-09-10T20:21:58.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-10T20:21:54.688942+0000 29'7 45:83 [1,2,0] 1 [1,2,0] 1 29'7 2024-09-10T20:21:54.688834+0000 29'7 2024-09-10T20:21:54.688834+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:21:54.688834+0000 4 0 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.425308+0000 0'0 45:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.928760+0000 0'0 45:30 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.928679+0000 0'0 45:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:21:59.035 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:59.036 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:21:59.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:21:59.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-10T20:21:59.048 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:21:54.687+0000 7f6fb517d640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-10T20:21:59.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-09-10T20:21:59.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-10T20:21:59.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-09-10T20:21:59.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-09-10T20:21:59.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-09-10T20:21:59.052 INFO:tasks.workunit.client.0.smithi181.stdout:prev count: 1 2024-09-10T20:21:59.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-10T20:21:59.144 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:21:59.144 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:21:59.144 INFO:tasks.workunit.client.0.smithi181.stdout: "must": true, 2024-09-10T20:21:59.144 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "0.000000" 2024-09-10T20:21:59.145 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:21:59.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-09-10T20:22:04.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:1.3 4 0 0 0 0 22 0 0 7 0 7 active+clean 2024-09-10T20:21:59.640634+0000 29'7 45:91 [1,2,0] 1 [1,2,0] 1 29'7 2024-09-10T20:21:59.640572+0000 29'7 2024-09-10T20:21:59.640572+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:21:59.640572+0000 4 0 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.425308+0000 0'0 45:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.928760+0000 0'0 45:30 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:21:49.928679+0000 0'0 45:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:20:54.087238+0000 0'0 2024-09-10T20:20:54.087238+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:54.087238+0000 0 0 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:22:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:22:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:22:04.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:22:04.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-10T20:22:04.435 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:21:54.687+0000 7f6fb517d640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-10T20:22:04.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-10T20:22:04.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-09-10T20:22:04.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-09-10T20:22:04.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-10T20:22:04.438 INFO:tasks.workunit.client.0.smithi181.stdout:current count: 1 2024-09-10T20:22:04.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-09-10T20:22:04.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-09-10T20:22:04.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-10T20:22:04.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:04.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:04.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:04.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:04.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:04.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:04.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-10T20:22:04.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-10T20:22:04.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-10T20:22:04.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-09-10T20:22:04.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:22:05.802 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:22:05.803 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-10T20:22:05.803 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-10T20:22:05.803 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-10T20:22:05.803 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:22:05.803 INFO:tasks.workunit.client.0.smithi181.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-10T20:22:05.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-10T20:22:05.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-10T20:22:05.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-10T20:22:05.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-09-10T20:22:06.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:06.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:06.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:06.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:06.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:06.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:06.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:06.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:06.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:06.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:06.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:06.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:06.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:06.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:22:06.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-10T20:22:06.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:06.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:06.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:22:06.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27299 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-mapper 0 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:06.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:06.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:06.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:06.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:06.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:06.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:06.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:06.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:06.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:06.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:06.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:06.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:06.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:06.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:06.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:07.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:22:07.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-10T20:22:07.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:07.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27299 2024-09-10T20:22:07.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27299 2024-09-10T20:22:07.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:07.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:07.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:22:07.005 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:22:07.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:22:07.073 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-09-10T20:22:07.073 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-09-10T20:22:07.137 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:22:07.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-recovery-scrub 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-09-10T20:22:07.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none ' 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-09-10T20:22:07.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:22:07.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-09-10T20:22:07.146 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_recovery_scrub_2' 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.43811 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:07.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:07.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:07.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:07.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:07.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:07.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:07.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:22:07.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-10T20:22:07.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:07.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:07.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43811 2024-09-10T20:22:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2024-09-10T20:22:07.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:22:07.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:07.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.43811 2024-09-10T20:22:07.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:22:07.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2024-09-10T20:22:07.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T20:22:07.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:22:07.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:07.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T20:22:07.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:22:07.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.43811/ceph-mon.a.asok 2024-09-10T20:22:07.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:22:07.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.43811/ceph-mon.a.asok config get fsid 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:22:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.43811/ceph-mon.a.asok 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:22:07.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.43811/ceph-mon.a.asok config get mon_host 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-10T20:22:07.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:22:08.757 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:22:08.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:08.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:08.758 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:08.760 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.760 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:08.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:08.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:22:08.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:22:08.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-09-10T20:22:08.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-09-10T20:22:08.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-09-10T20:22:08.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-10T20:22:08.789 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-09-10T20:22:08.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-10T20:22:08.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:08.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12359fff-513f-44a8-b5bb-83d355f190a8 2024-09-10T20:22:08.797 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 12359fff-513f-44a8-b5bb-83d355f190a8 2024-09-10T20:22:08.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 12359fff-513f-44a8-b5bb-83d355f190a8' 2024-09-10T20:22:08.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:08.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwquBmUHc6MBAA75OQgKrsw7vpZnV29VxPeg== 2024-09-10T20:22:08.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwquBmUHc6MBAA75OQgKrsw7vpZnV29VxPeg=="}' 2024-09-10T20:22:08.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12359fff-513f-44a8-b5bb-83d355f190a8 -i td/osd-recovery-scrub/0/new.json 2024-09-10T20:22:09.013 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:22:09.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-10T20:22:09.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBwquBmUHc6MBAA75OQgKrsw7vpZnV29VxPeg== --osd-uuid 12359fff-513f-44a8-b5bb-83d355f190a8 2024-09-10T20:22:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:09.053+0000 7f617b63b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:09.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:09.061+0000 7f617b63b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:09.062+0000 7f617b63b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:09.063+0000 7f617b63b640 -1 bdev(0x55ffe52b7c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:09.063+0000 7f617b63b640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-10T20:22:14.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-10T20:22:14.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:14.367 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:22:14.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:22:14.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:14.723 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:22:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:22:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-10T20:22:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:14.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:14.748+0000 7fb649989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:14.763 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:14.760+0000 7fb649989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:14.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:14.761+0000 7fb649989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:15.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:22:15.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:15.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:22:15.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:15.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:15.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:15.012 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:22:15.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:15.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:15.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:15.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:16.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:16.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:16.287 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:22:16.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:16.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:16.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:16.614 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:16.610+0000 7fb649989640 -1 Falling back to public interface 2024-09-10T20:22:17.573 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:22:17.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:17.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:17.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:17.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:17.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:17.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:18.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:18.268+0000 7fb649989640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:22:18.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:18.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:18.855 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:22:18.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:18.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:18.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:19.165 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1440508934,v1:127.0.0.1:6803/1440508934] [v2:127.0.0.1:6804/1440508934,v1:127.0.0.1:6805/1440508934] exists,up 12359fff-513f-44a8-b5bb-83d355f190a8 2024-09-10T20:22:19.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:22:19.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:22:19.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-10T20:22:19.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:19.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12e78397-f06d-4b4e-9b15-f65775ff5f83 2024-09-10T20:22:19.171 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 12e78397-f06d-4b4e-9b15-f65775ff5f83 2024-09-10T20:22:19.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 12e78397-f06d-4b4e-9b15-f65775ff5f83' 2024-09-10T20:22:19.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:19.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7quBm09HhChAAOVeSC79h97/HOcVaw9q1Jw== 2024-09-10T20:22:19.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7quBm09HhChAAOVeSC79h97/HOcVaw9q1Jw=="}' 2024-09-10T20:22:19.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12e78397-f06d-4b4e-9b15-f65775ff5f83 -i td/osd-recovery-scrub/1/new.json 2024-09-10T20:22:19.532 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:22:19.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-10T20:22:19.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB7quBm09HhChAAOVeSC79h97/HOcVaw9q1Jw== --osd-uuid 12e78397-f06d-4b4e-9b15-f65775ff5f83 2024-09-10T20:22:19.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:19.570+0000 7fa57fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:19.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:19.621+0000 7fa57fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:19.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:19.622+0000 7fa57fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:19.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:19.622+0000 7fa57fc2c640 -1 bdev(0x55a66ea37c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:19.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:19.622+0000 7fa57fc2c640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-10T20:22:25.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-10T20:22:25.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:25.648 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:22:25.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:22:25.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:26.007 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:22:26.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:22:26.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-10T20:22:26.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:26.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:26.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:26.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:26.031+0000 7f7ad3e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:26.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:26.033+0000 7f7ad3e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:26.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:26.034+0000 7f7ad3e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:26.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:26.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:26.283 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:22:26.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:26.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:26.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:27.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:27.399+0000 7f7ad3e33640 -1 Falling back to public interface 2024-09-10T20:22:27.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:27.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:27.559 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:22:27.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:27.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:27.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:27.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:28.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:28.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:28.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:28.842 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:22:28.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:28.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:29.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:29.118+0000 7f7ad3e33640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:22:29.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:30.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:30.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:30.147 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:22:30.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:30.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:30.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:30.478 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/784296863,v1:127.0.0.1:6811/784296863] [v2:127.0.0.1:6812/784296863,v1:127.0.0.1:6813/784296863] exists,up 12e78397-f06d-4b4e-9b15-f65775ff5f83 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-09-10T20:22:30.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-10T20:22:30.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:30.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-10T20:22:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-10T20:22:30.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=00c73b3d-57fd-432c-9f4b-3af1bccbb577 2024-09-10T20:22:30.486 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 00c73b3d-57fd-432c-9f4b-3af1bccbb577 2024-09-10T20:22:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 00c73b3d-57fd-432c-9f4b-3af1bccbb577' 2024-09-10T20:22:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:30.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGquBm5PmtHRAA8LOCqDdoiU1umTjCx9DrBw== 2024-09-10T20:22:30.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGquBm5PmtHRAA8LOCqDdoiU1umTjCx9DrBw=="}' 2024-09-10T20:22:30.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 00c73b3d-57fd-432c-9f4b-3af1bccbb577 -i td/osd-recovery-scrub/2/new.json 2024-09-10T20:22:30.908 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:22:30.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-10T20:22:30.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCGquBm5PmtHRAA8LOCqDdoiU1umTjCx9DrBw== --osd-uuid 00c73b3d-57fd-432c-9f4b-3af1bccbb577 2024-09-10T20:22:30.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:30.947+0000 7fd475f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:31.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:31.016+0000 7fd475f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:31.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:31.017+0000 7fd475f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:31.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:31.018+0000 7fd475f02640 -1 bdev(0x55ebcdf19c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:31.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:31.018+0000 7fd475f02640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-10T20:22:37.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-10T20:22:37.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:37.181 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:22:37.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:22:37.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:37.544 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:22:37.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:22:37.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-10T20:22:37.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:37.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:37.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:37.569+0000 7f4b9860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:37.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:37.571+0000 7f4b9860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:37.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:37.572+0000 7f4b9860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:37.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:22:37.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:37.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:37.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:37.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:22:38.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:38.688 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:38.685+0000 7f4b9860d640 -1 Falling back to public interface 2024-09-10T20:22:39.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:39.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:39.100 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:22:39.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:39.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:39.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:22:39.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:40.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:40.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:40.387 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:22:40.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:40.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:40.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:22:40.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:40.503+0000 7f4b9860d640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:22:40.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:41.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:41.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:41.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:41.708 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:22:41.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:41.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:22:41.985 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2424467169,v1:127.0.0.1:6819/2424467169] [v2:127.0.0.1:6820/2424467169,v1:127.0.0.1:6821/2424467169] exists,up 00c73b3d-57fd-432c-9f4b-3af1bccbb577 2024-09-10T20:22:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:22:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:22:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:22:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:22:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-09-10T20:22:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-10T20:22:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:22:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:41.992 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e4cf4201-6b46-417f-8d8d-cbcf185d2bfa 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e4cf4201-6b46-417f-8d8d-cbcf185d2bfa' 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 e4cf4201-6b46-417f-8d8d-cbcf185d2bfa 2024-09-10T20:22:41.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:42.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCSquBmOt9DABAA/f1G0BvE6gxaolLqYq5JYg== 2024-09-10T20:22:42.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCSquBmOt9DABAA/f1G0BvE6gxaolLqYq5JYg=="}' 2024-09-10T20:22:42.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e4cf4201-6b46-417f-8d8d-cbcf185d2bfa -i td/osd-recovery-scrub/3/new.json 2024-09-10T20:22:42.393 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:22:42.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-10T20:22:42.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCSquBmOt9DABAA/f1G0BvE6gxaolLqYq5JYg== --osd-uuid e4cf4201-6b46-417f-8d8d-cbcf185d2bfa 2024-09-10T20:22:42.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:42.431+0000 7fe5eb7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:42.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:42.474+0000 7fe5eb7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:42.478 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:42.475+0000 7fe5eb7db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:42.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:42.475+0000 7fe5eb7db640 -1 bdev(0x561e4ad5bc00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:42.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:42.475+0000 7fe5eb7db640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-10T20:22:48.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-10T20:22:48.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:48.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:22:48.216 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T20:22:48.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:48.577 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T20:22:48.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:22:48.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=d47ff1c2-097a-49e3-a94c-917e2e64e57c --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43811/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-10T20:22:48.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:48.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:48.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:48.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:48.602+0000 7f3f85411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:48.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:48.604+0000 7f3f85411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:48.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:48.605+0000 7f3f85411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:22:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:22:48.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:48.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:48.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:48.857 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:22:48.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:48.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:48.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:22:49.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:50.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:50.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:50.131 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:22:50.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:50.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:50.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:22:50.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:50.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:50.461+0000 7f3f85411640 -1 Falling back to public interface 2024-09-10T20:22:51.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:51.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:51.415 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:22:51.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:51.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:51.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:22:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:52.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:22:52.538+0000 7f3f85411640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:22:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:52.700 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:22:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:22:53.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:54.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:54.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:54.095 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:22:54.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:22:54.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:54.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:22:54.378 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2506541487,v1:127.0.0.1:6827/2506541487] [v2:127.0.0.1:6828/2506541487,v1:127.0.0.1:6829/2506541487] exists,up e4cf4201-6b46-417f-8d8d-cbcf185d2bfa 2024-09-10T20:22:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:22:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:22:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:22:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-09-10T20:22:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:22:54.770 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T20:22:54.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:22:55.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-09-10T20:22:55.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:22:55.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:22:55.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:22:55.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:22:55.785 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:22:55.785 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:22:55.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:22:55.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:22:55.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:22:55.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:22:56.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:22:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:22:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:22:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:22:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:22:56.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:22:56.156 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:22:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-10T20:22:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-10T20:22:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-10T20:22:56.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:22:56.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:22:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-10T20:22:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-10T20:22:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-09-10T20:22:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:22:56.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:22:56.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-10T20:22:56.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-10T20:22:56.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444' 2024-09-10T20:22:56.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:22:56.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:22:56.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345922 2024-09-10T20:22:56.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345922 2024-09-10T20:22:56.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444 3-85899345922' 2024-09-10T20:22:56.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:22:56.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-10T20:22:56.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:22:56.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:22:56.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-10T20:22:56.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:22:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-10T20:22:56.568 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836489 2024-09-10T20:22:56.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-10T20:22:56.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:22:56.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-10T20:22:56.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:22:57.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:22:57.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:22:58.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-10T20:22:58.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:22:58.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-10T20:22:58.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:22:58.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:22:58.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-10T20:22:58.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:22:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-10T20:22:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-10T20:22:58.135 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672967 2024-09-10T20:22:58.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:22:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-10T20:22:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:22:58.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-10T20:22:58.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:22:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:22:58.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-10T20:22:58.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:22:58.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-10T20:22:58.416 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509444 2024-09-10T20:22:58.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-10T20:22:58.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:22:58.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-10T20:22:58.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:22:58.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345922 2024-09-10T20:22:58.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:22:58.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:22:58.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345922 2024-09-10T20:22:58.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:22:58.844 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345922 2024-09-10T20:22:58.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345922 2024-09-10T20:22:58.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-09-10T20:22:58.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:22:59.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345922 -lt 85899345922 2024-09-10T20:22:59.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:22:59.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:22:59.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:22:59.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:22:59.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:22:59.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:22:59.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:22:59.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:22:59.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:23:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:23:00.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:23:00.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:23:00.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-09-10T20:23:00.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-09-10T20:23:00.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-09-10T20:23:00.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-09-10T20:23:00.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-10T20:23:00.639 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:23:00.639 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:22:55.463738+0000 0'0 23:9 [1] 1 [1] 1 0'0 2024-09-10T20:22:54.712684+0000 0'0 2024-09-10T20:22:54.712684+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:54:45.838668+0000 0 0 2024-09-10T20:23:00.639 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:23:00.639 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:23:00.639 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:23:00.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.43811 bs=1M count=50 2024-09-10T20:23:00.791 INFO:tasks.workunit.client.0.smithi181.stderr:50+0 records in 2024-09-10T20:23:00.792 INFO:tasks.workunit.client.0.smithi181.stderr:50+0 records out 2024-09-10T20:23:00.792 INFO:tasks.workunit.client.0.smithi181.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.140446 s, 373 MB/s 2024-09-10T20:23:00.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-09-10T20:23:00.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:00.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.43811 2024-09-10T20:23:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:02.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.43811 2024-09-10T20:23:03.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:03.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.43811 2024-09-10T20:23:05.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:05.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.43811 2024-09-10T20:23:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.43811 2024-09-10T20:23:08.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:08.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.43811 2024-09-10T20:23:10.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:10.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.43811 2024-09-10T20:23:11.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:11.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.43811 2024-09-10T20:23:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:13.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.43811 2024-09-10T20:23:14.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:14.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.43811 2024-09-10T20:23:15.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:15.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.43811 2024-09-10T20:23:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.43811 2024-09-10T20:23:18.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:18.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.43811 2024-09-10T20:23:20.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:20.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.43811 2024-09-10T20:23:22.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:22.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.43811 2024-09-10T20:23:23.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:23.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.43811 2024-09-10T20:23:25.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:25.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.43811 2024-09-10T20:23:26.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:26.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.43811 2024-09-10T20:23:28.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:28.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.43811 2024-09-10T20:23:29.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:29.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.43811 2024-09-10T20:23:31.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:31.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.43811 2024-09-10T20:23:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:32.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.43811 2024-09-10T20:23:34.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:34.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.43811 2024-09-10T20:23:36.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:36.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.43811 2024-09-10T20:23:37.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:37.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.43811 2024-09-10T20:23:39.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:39.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.43811 2024-09-10T20:23:40.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:40.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.43811 2024-09-10T20:23:42.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:42.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.43811 2024-09-10T20:23:43.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:43.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.43811 2024-09-10T20:23:45.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:45.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.43811 2024-09-10T20:23:46.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:46.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.43811 2024-09-10T20:23:48.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:48.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.43811 2024-09-10T20:23:49.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:49.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.43811 2024-09-10T20:23:51.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:51.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.43811 2024-09-10T20:23:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.43811 2024-09-10T20:23:54.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:54.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.43811 2024-09-10T20:23:55.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:55.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.43811 2024-09-10T20:23:57.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:57.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.43811 2024-09-10T20:23:58.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:23:58.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.43811 2024-09-10T20:24:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:00.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.43811 2024-09-10T20:24:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:02.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.43811 2024-09-10T20:24:03.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:03.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.43811 2024-09-10T20:24:04.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:04.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.43811 2024-09-10T20:24:06.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:06.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.43811 2024-09-10T20:24:07.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:07.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.43811 2024-09-10T20:24:09.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:09.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.43811 2024-09-10T20:24:10.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:10.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.43811 2024-09-10T20:24:12.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:12.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.43811 2024-09-10T20:24:13.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:13.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.43811 2024-09-10T20:24:15.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:15.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.43811 2024-09-10T20:24:16.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:16.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.43811 2024-09-10T20:24:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.43811 2024-09-10T20:24:19.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:19.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.43811 2024-09-10T20:24:21.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:21.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.43811 2024-09-10T20:24:22.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:22.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.43811 2024-09-10T20:24:24.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:24.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.43811 2024-09-10T20:24:25.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:25.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.43811 2024-09-10T20:24:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:26.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.43811 2024-09-10T20:24:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:28.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.43811 2024-09-10T20:24:29.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:29.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.43811 2024-09-10T20:24:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.43811 2024-09-10T20:24:33.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:33.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.43811 2024-09-10T20:24:34.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:34.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.43811 2024-09-10T20:24:35.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:35.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.43811 2024-09-10T20:24:37.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:37.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.43811 2024-09-10T20:24:38.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:38.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.43811 2024-09-10T20:24:40.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:40.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.43811 2024-09-10T20:24:41.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:41.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.43811 2024-09-10T20:24:43.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.43811 2024-09-10T20:24:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.43811 2024-09-10T20:24:46.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:46.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.43811 2024-09-10T20:24:47.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:47.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.43811 2024-09-10T20:24:49.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:49.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.43811 2024-09-10T20:24:50.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:50.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.43811 2024-09-10T20:24:52.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:52.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.43811 2024-09-10T20:24:53.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:53.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.43811 2024-09-10T20:24:55.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:55.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.43811 2024-09-10T20:24:56.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:56.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.43811 2024-09-10T20:24:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.43811 2024-09-10T20:24:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:24:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.43811 2024-09-10T20:25:01.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:01.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.43811 2024-09-10T20:25:02.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:02.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.43811 2024-09-10T20:25:04.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:04.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.43811 2024-09-10T20:25:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.43811 2024-09-10T20:25:07.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:07.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.43811 2024-09-10T20:25:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.43811 2024-09-10T20:25:10.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:10.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.43811 2024-09-10T20:25:11.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:11.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.43811 2024-09-10T20:25:13.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:13.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.43811 2024-09-10T20:25:14.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:14.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.43811 2024-09-10T20:25:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.43811 2024-09-10T20:25:17.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.43811 2024-09-10T20:25:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.43811 2024-09-10T20:25:20.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:20.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.43811 2024-09-10T20:25:22.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:22.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.43811 2024-09-10T20:25:23.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:23.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.43811 2024-09-10T20:25:25.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:25.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.43811 2024-09-10T20:25:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.43811 2024-09-10T20:25:28.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:28.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.43811 2024-09-10T20:25:29.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-10T20:25:29.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.43811 2024-09-10T20:25:31.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.43811 2024-09-10T20:25:31.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-09-10T20:25:31.579 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 4 2024-09-10T20:25:31.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-09-10T20:25:31.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-09-10T20:25:31.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-10T20:25:31.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-10T20:25:31.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-10T20:25:31.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-10T20:25:31.854 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:25:31.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-10T20:25:33.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-09-10T20:25:33.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-09-10T20:25:33.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-09-10T20:25:33.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-10T20:25:33.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-10T20:25:33.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-10T20:25:33.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-10T20:25:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:25:34.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-10T20:25:36.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-09-10T20:25:36.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-09-10T20:25:36.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-09-10T20:25:36.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-10T20:25:36.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-10T20:25:36.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-10T20:25:36.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-10T20:25:36.620 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:25:36.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-10T20:25:38.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-09-10T20:25:38.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-09-10T20:25:38.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-09-10T20:25:38.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-10T20:25:38.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-10T20:25:38.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-10T20:25:38.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-10T20:25:38.915 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:25:38.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-09-10T20:25:38.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-09-10T20:25:38.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-10T20:25:39.201 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:25:39.201 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-09-10T20:25:35.280710+0000 24'1300 27:1324 [1,0,2,3] 1 [1,0] 1 0'0 2024-09-10T20:22:54.712684+0000 0'0 2024-09-10T20:22:54.712684+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:25:39.201 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:25:39.201 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:25:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:25:39.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-09-10T20:25:49.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-09-10T20:25:49.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:25:49.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:25:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:25:49.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:25:49.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:25:50.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:25:50.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-09-10T20:25:50.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2024-09-10T20:25:50.526 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-09-10T20:25:50.527 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-09-10T20:25:50.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-10T20:25:50.531 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:40.800+0000 7fb642925640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.531 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:43.808+0000 7fb642925640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.531 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:44.814+0000 7fb642925640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.531 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:46.866+0000 7fb642925640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-10T20:25:50.547 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:37.297+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.547 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:38.269+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.547 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:43.143+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.548 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:45.154+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.548 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:46.204+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.548 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:48.165+0000 7f7accdcf640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-10T20:25:50.549 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:38.132+0000 7f4b9103e640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.550 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:43.999+0000 7f4b9103e640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.550 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:47.987+0000 7f4b9103e640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.550 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:48.982+0000 7f4b9103e640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-10T20:25:50.553 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:36.750+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.553 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:40.678+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.553 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:42.680+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.553 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:43.692+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.553 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:44.710+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.554 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:25:48.763+0000 7f3f7dd9f640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-10T20:25:50.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-09-10T20:25:50.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-09-10T20:25:50.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-09-10T20:25:50.555 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-09-10T20:25:50.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-09-10T20:25:50.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-10T20:25:50.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-10T20:25:50.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-09-10T20:25:50.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-09-10T20:25:50.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-10T20:25:50.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-10T20:25:50.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-09-10T20:25:50.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-09-10T20:25:50.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-10T20:25:50.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-10T20:25:50.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-09-10T20:25:50.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-09-10T20:25:50.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:25:50.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-10T20:25:50.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-10T20:25:50.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-09-10T20:25:50.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-09-10T20:25:50.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-09-10T20:25:50.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:25:50.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:25:50.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:25:50.589 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:25:50.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:25:50.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:25:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:25:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:25:50.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:25:50.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:25:50.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:25:50.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:25:50.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:25:50.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-10T20:25:50.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:25:50.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:50.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:25:50.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43811 2024-09-10T20:25:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:25:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:25:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-09-10T20:25:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-09-10T20:25:50.618 INFO:tasks.workunit.client.0.smithi181.stdout:TEST PASSED 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: local poolname=test 2024-09-10T20:25:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: return 0 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-recovery-scrub 0 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:25:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:25:50.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:25:50.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:25:50.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:25:50.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:25:50.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:25:50.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:25:50.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:25:50.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-10T20:25:50.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:25:50.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:50.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43811 2024-09-10T20:25:50.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43811 2024-09-10T20:25:50.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:25:50.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:25:50.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:25:50.629 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:25:50.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:25:50.695 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-09-10T20:25:50.695 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-09-10T20:25:50.758 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:25:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-dump 2024-09-10T20:25:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-09-10T20:25:50.765 INFO:tasks.workunit.client.0.smithi181.stdout:This test is disabled 2024-09-10T20:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-09-10T20:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-dump 0 2024-09-10T20:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:25:50.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:25:50.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:25:50.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:25:50.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:25:50.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:25:50.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:25:50.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:25:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2024-09-10T20:25:50.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:25:50.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:50.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52394 2024-09-10T20:25:50.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52394 2024-09-10T20:25:50.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:25:50.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:25:50.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:25:50.775 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:25:50.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:25:50.839 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-09-10T20:25:50.839 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-09-10T20:25:50.899 INFO:tasks.workunit.client.0.smithi181.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-10T20:25:50.899 INFO:tasks.workunit.client.0.smithi181.stderr:++ TIMEOUT=300 2024-09-10T20:25:50.899 INFO:tasks.workunit.client.0.smithi181.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:++ MAX_TIMEOUT=15 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:++ PG_NUM=4 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:++ TMPDIR=/tmp 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-10T20:25:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:+++ uname 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:++ '[' Linux = FreeBSD ']' 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:++ SED=sed 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:++ AWK=awk 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:+++ stty -a 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:+++ head -1 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-10T20:25:50.901 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:25:50.902 INFO:tasks.workunit.client.0.smithi181.stderr:++ termwidth= 2024-09-10T20:25:50.902 INFO:tasks.workunit.client.0.smithi181.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-10T20:25:50.902 INFO:tasks.workunit.client.0.smithi181.stderr:++ DIFFCOLOPTS='-y ' 2024-09-10T20:25:50.902 INFO:tasks.workunit.client.0.smithi181.stderr:++ KERNCORE=kernel.core_pattern 2024-09-10T20:25:50.902 INFO:tasks.workunit.client.0.smithi181.stderr:++ EXTRA_OPTS= 2024-09-10T20:25:50.905 INFO:tasks.workunit.client.0.smithi181.stderr:++ test '' = TESTS 2024-09-10T20:25:50.905 INFO:tasks.workunit.client.0.smithi181.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-09-10T20:25:50.905 INFO:tasks.workunit.client.0.smithi181.stderr:++ uname 2024-09-10T20:25:50.906 INFO:tasks.workunit.client.0.smithi181.stderr:+ '[' Linux = FreeBSD ']' 2024-09-10T20:25:50.906 INFO:tasks.workunit.client.0.smithi181.stderr:+ use_ec_overwrite=true 2024-09-10T20:25:50.906 INFO:tasks.workunit.client.0.smithi181.stderr:+ getjson=no 2024-09-10T20:25:50.906 INFO:tasks.workunit.client.0.smithi181.stderr:+ jqfilter='def walk(f): 2024-09-10T20:25:50.906 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T20:25:50.907 INFO:tasks.workunit.client.0.smithi181.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T20:25:50.912 INFO:tasks.workunit.client.0.smithi181.stderr:+ main osd-scrub-repair 2024-09-10T20:25:50.912 INFO:tasks.workunit.client.0.smithi181.stderr:+ local dir=td/osd-scrub-repair 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:+ shift 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:+ shopt -s -o xtrace 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:25:50.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-repair 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none ' 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-09-10T20:25:50.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:25:50.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-09-10T20:25:50.916 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_bluestore_basic 2024-09-10T20:25:50.916 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_bluestore_failed 2024-09-10T20:25:50.916 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_bluestore_scrub 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_bluestore_tag 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_erasure_coded_appends 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_and_repair_lrc_appends 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_and_repair_replicated 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_scrub_erasure_appends 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-09-10T20:25:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_scrub_replicated 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_corrupt_snapset_scrub_rep 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_list_missing_erasure_coded_appends 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_list_missing_erasure_coded_overwrites 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_periodic_scrub_replicated 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_repair_stats 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_repair_stats_ec 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_request_scrub_priority 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_warning 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_skip_non_repair_during_recovery 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_unfound_erasure_coded_appends 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_unfound_erasure_coded_overwrites' 2024-09-10T20:25:50.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:25:50.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:25:50.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:25:50.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:25:50.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:25:50.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:25:50.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:25:50.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:50.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:50.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:25:50.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:25:50.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:25:50.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:25:50.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:25:50.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:50.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:50.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:25:50.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:25:50.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:25:50.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:25:50.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:25:50.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:25:50.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-10T20:25:51.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:25:51.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:25:51.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T20:25:51.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:25:51.616 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:51.617 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:51.617 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:25:51.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:25:51.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:51.696 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:25:51.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:25:51.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:25:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-10T20:25:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:25:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:25:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:25:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:25:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:25:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:52.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:25:52.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:25:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:25:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:25:52.623 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:25:52.624 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:25:52.624 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:25:52.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:25:52.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:25:52.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:25:52.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:25:52.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 2024-09-10T20:25:52.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a61a79a4-ba0a-4322-9f4f-12ae1e5bba45' 2024-09-10T20:25:52.626 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 2024-09-10T20:25:52.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:25:52.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQq+Bm+GgGJhAA5beIphvkzHfZ/IKe+D2JoA== 2024-09-10T20:25:52.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQq+Bm+GgGJhAA5beIphvkzHfZ/IKe+D2JoA=="}' 2024-09-10T20:25:52.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:25:52.847 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:25:52.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:25:52.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBQq+Bm+GgGJhAA5beIphvkzHfZ/IKe+D2JoA== --osd-uuid a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 2024-09-10T20:25:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:52.884+0000 7f9509ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:52.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:52.893+0000 7f9509ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:52.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:52.894+0000 7f9509ca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:52.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:52.895+0000 7f9509ca5640 -1 bdev(0x56085047dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:52.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:52.895+0000 7f9509ca5640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:25:59.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:25:59.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:25:59.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:25:59.174 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:25:59.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:25:59.530 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:25:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:25:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T20:25:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:25:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:25:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:25:59.560 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:59.556+0000 7fe21de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:59.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:59.558+0000 7fe21de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:59.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:25:59.559+0000 7fe21de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:59.809 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:25:59.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:25:59.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:59.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:00.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:00.906 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:00.903+0000 7fe21de0d640 -1 Falling back to public interface 2024-09-10T20:26:01.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:01.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:01.083 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:26:01.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:26:01.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:01.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:01.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:02.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:02.371 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:26:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:26:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:02.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:02.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:02.475+0000 7fe21de0d640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:26:02.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:03.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:03.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:03.663 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:26:03.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:26:03.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:03.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:03.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:04.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:04.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:04.952 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:26:04.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:26:04.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:04.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:05.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:06.249 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:06.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:07.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:07.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:07.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:26:07.539 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:26:07.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:07.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:07.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:08.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:08.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:08.823 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:26:08.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:26:08.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:08.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:09.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:10.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:10.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:10.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:26:10.233 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:26:10.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:10.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:10.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:11.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:11.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:11.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:26:11.524 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:26:11.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:11.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:11.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:12.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:26:12.811 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:26:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:13.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:14.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:14.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:14.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:26:14.098 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:26:14.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:14.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:14.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:15.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:15.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:15.723 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:26:15.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:26:15.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:15.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:16.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:17.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:17.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:17.242 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:26:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:26:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:17.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:17.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:18.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:18.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:18.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:26:18.849 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:26:18.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:18.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:19.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:20.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:20.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:20.618 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:26:20.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:26:20.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:20.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:21.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:22.345 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:26:22.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:26:22.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:22.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:22.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:23.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:23.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:23.873 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:26:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:26:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:23.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:24.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:25.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:25.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:25.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:26:25.167 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:26:25.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:25.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:25.449 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1857085732,v1:127.0.0.1:6803/1857085732] [v2:127.0.0.1:6804/1857085732,v1:127.0.0.1:6805/1857085732] exists,up a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:26:25.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:26:25.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:26:25.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5af9e73f-23ad-4d56-953f-37d6f351f814 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5af9e73f-23ad-4d56-953f-37d6f351f814' 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 5af9e73f-23ad-4d56-953f-37d6f351f814 2024-09-10T20:26:25.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:26:25.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBxq+Bmf3XrGxAACAE1tRH+QmpyCbA/xlX+CQ== 2024-09-10T20:26:25.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBxq+Bmf3XrGxAACAE1tRH+QmpyCbA/xlX+CQ=="}' 2024-09-10T20:26:25.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5af9e73f-23ad-4d56-953f-37d6f351f814 -i td/osd-scrub-repair/1/new.json 2024-09-10T20:26:25.884 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:26:25.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:26:25.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBxq+Bmf3XrGxAACAE1tRH+QmpyCbA/xlX+CQ== --osd-uuid 5af9e73f-23ad-4d56-953f-37d6f351f814 2024-09-10T20:26:25.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:25.924+0000 7f0064dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:25.968 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:25.965+0000 7f0064dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:25.969 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:25.966+0000 7f0064dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:25.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:25.966+0000 7f0064dcc640 -1 bdev(0x55ea10ec3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:26:25.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:25.966+0000 7f0064dcc640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:26:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:26:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:26:31.970 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:26:31.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:26:31.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:26:32.325 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:26:32.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:26:32.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T20:26:32.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:26:32.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:26:32.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:26:32.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:32.351+0000 7f91966b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:32.357 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:32.353+0000 7f91966b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:32.355+0000 7f91966b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:26:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:26:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:26:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:32.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:32.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:33.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:33.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:33.903 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:26:33.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:26:33.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:33.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:34.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:34.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:34.193+0000 7f91966b6640 -1 Falling back to public interface 2024-09-10T20:26:35.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:35.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:35.190 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:26:35.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:26:35.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:35.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:35.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:36.296 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:26:36.292+0000 7f91966b6640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:26:36.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:36.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:36.465 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:26:36.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:26:36.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:36.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:36.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:37.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:37.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:37.808 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:26:37.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:26:37.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:37.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:38.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:39.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:39.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:39.095 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:26:39.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:26:39.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:39.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:39.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:40.446 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:26:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:26:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:40.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:41.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:41.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:41.740 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:26:41.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:26:41.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:41.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:42.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:43.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:43.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:43.064 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:26:43.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:26:43.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:43.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:43.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:44.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:44.353 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:26:44.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:26:44.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:44.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:44.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:45.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:45.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:45.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:26:45.656 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:26:45.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:45.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:45.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:46.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:46.946 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:26:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:26:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:46.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:47.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:48.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:48.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:48.248 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:26:48.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:26:48.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:48.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:48.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:49.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:49.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:49.818 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:26:49.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:26:49.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:49.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:51.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:51.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:51.409 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:26:51.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:26:51.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:51.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:51.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:52.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:52.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:52.936 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:26:52.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:26:52.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:52.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:53.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:54.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:54.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:54.321 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:26:54.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:26:54.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:54.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:55.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:56.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:56.332 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:26:56.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:26:56.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:56.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:56.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:57.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:57.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:57.884 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:26:57.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:26:57.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:57.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:59.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:59.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:59.174 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:26:59.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:26:59.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:59.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:26:59.453 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3038253741,v1:127.0.0.1:6811/3038253741] [v2:127.0.0.1:6812/3038253741,v1:127.0.0.1:6813/3038253741] exists,up 5af9e73f-23ad-4d56-953f-37d6f351f814 2024-09-10T20:26:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:26:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:26:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:26:59.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-09-10T20:26:59.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T20:26:59.811 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T20:26:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T20:26:59.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T20:27:00.188 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T20:27:00.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:27:01.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T20:27:03.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-09-10T20:27:03.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:27:03.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:27:03.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:27:03.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:27:03.096 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:27:03.096 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:27:03.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:27:03.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:27:03.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:27:03.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:27:03.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:27:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T20:27:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:27:03.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:03.468 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:27:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-10T20:27:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-10T20:27:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-10T20:27:03.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:03.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:27:03.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-10T20:27:03.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-10T20:27:03.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2024-09-10T20:27:03.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:03.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-10T20:27:03.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:03.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:27:03.675 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-10T20:27:03.675 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:03.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-10T20:27:03.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-10T20:27:03.677 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836490 2024-09-10T20:27:03.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:03.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-10T20:27:03.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:27:04.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:27:04.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:05.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-10T20:27:05.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:05.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-10T20:27:05.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:05.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:27:05.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-10T20:27:05.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:05.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-10T20:27:05.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-10T20:27:05.239 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672969 2024-09-10T20:27:05.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:27:05.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-10T20:27:05.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:27:05.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:05.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:05.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:27:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:27:05.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:27:05.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:27:05.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:27:05.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:27:05.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:27:05.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:27:06.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:27:06.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:27:06.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:06.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T20:27:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T20:27:06.952 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T20:27:06.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T20:27:07.393 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T20:27:07.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T20:27:07.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T20:27:07.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:27:07.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-10T20:27:07.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-10T20:27:07.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:27:07.677 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-10T20:27:07.678 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T20:27:07.678 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:27:07.678 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T20:27:07.678 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:27:08.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:27:08.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T20:27:08.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:27:08.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-10T20:27:08.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T20:27:08.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:27:08.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:27:08.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:27:08.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:27:08.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:27:09.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T20:27:11.138 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:27:11.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:27:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:27:11.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:27:11.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:27:11.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:27:11.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:27:11.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:27:11.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:11.684+0000 7fb881c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:11.689 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:11.686+0000 7fb881c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:11.691 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:11.687+0000 7fb881c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:27:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:11.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:27:12.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:13.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:13.029+0000 7fb881c0e640 -1 Falling back to public interface 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:13.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:27:13.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:14.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:27:14.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:14.747+0000 7fb881c0e640 -1 osd.0 22 log_to_monitors true 2024-09-10T20:27:14.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:15.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:15.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:15.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:27:15.838 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:27:15.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:15.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/3840716951,v1:127.0.0.1:6803/3840716951] [v2:127.0.0.1:6804/3840716951,v1:127.0.0.1:6805/3840716951] exists,up a61a79a4-ba0a-4322-9f4f-12ae1e5bba45 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:27:16.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:27:16.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:27:16.488 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T20:27:16.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:27:16.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:16.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:27:16.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-10T20:27:16.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-10T20:27:16.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-10T20:27:16.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:16.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:27:16.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-10T20:27:16.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-10T20:27:16.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672973' 2024-09-10T20:27:16.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:16.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-10T20:27:16.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:16.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:27:16.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-10T20:27:16.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:16.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-10T20:27:16.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-10T20:27:16.698 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149698 2024-09-10T20:27:16.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:16.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-10T20:27:16.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:27:17.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:27:17.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-10T20:27:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:19.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-10T20:27:19.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:19.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-10T20:27:19.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:19.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:27:19.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-10T20:27:19.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:19.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-10T20:27:19.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-10T20:27:19.545 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672973 2024-09-10T20:27:19.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:27:19.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-10T20:27:19.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:27:19.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:19.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:20.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:27:20.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:27:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:27:20.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:27:20.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:20.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-10T20:27:20.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:27:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-10T20:27:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:27:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T20:27:21.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:21.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-10T20:27:21.588 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T20:27:21.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T20:27:21.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:27:21.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:27:21.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T20:27:21.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:27:00.138489+0000 '>' 2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:21.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:27:22.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T20:27:23.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:27:00.138489+0000 '>' 2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:23.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:27:24.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:27:24.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T20:27:24.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:27:00.138489+0000 '>' 2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:24.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:27:25.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:27:25.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:27:25.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T20:27:25.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T20:27:25.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:27:25.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:27:25.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T20:27:25.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:27:22.260376+0000 '>' 2024-09-10T20:27:00.138489+0000 2024-09-10T20:27:25.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:27:25.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:27:27.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:27:27.724 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:27:27.724 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:27:27.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:27:27.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:27:27.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:27:28.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:27:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:27:28.512 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T20:27:28.513 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:27:28.513 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:27:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:27:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:27:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:27:28.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:27:28.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:28.817+0000 7ffb59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:28.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:28.820+0000 7ffb59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:28.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:28.821+0000 7ffb59dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:29.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:30.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:30.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:30.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:27:30.355 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:27:30.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:30.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:30.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:30.442+0000 7ffb59dc5640 -1 Falling back to public interface 2024-09-10T20:27:30.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:31.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:31.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:32.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:32.715+0000 7ffb59dc5640 -1 osd.1 28 log_to_monitors true 2024-09-10T20:27:32.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:32.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:32.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:27:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:27:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:33.326 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 32 up_thru 26 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2947540639,v1:127.0.0.1:6811/2947540639] [v2:127.0.0.1:6812/2947540639,v1:127.0.0.1:6813/2947540639] exists,up 5af9e73f-23ad-4d56-953f-37d6f351f814 2024-09-10T20:27:33.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:27:33.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:27:33.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:27:33.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:27:33.712 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T20:27:33.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:27:33.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:33.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:27:33.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-10T20:27:33.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-10T20:27:33.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-10T20:27:33.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:27:33.868 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-10T20:27:34.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:34.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:27:34.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-10T20:27:34.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:34.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-10T20:27:34.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-10T20:27:34.014 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149702 2024-09-10T20:27:34.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:34.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-10T20:27:34.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:27:35.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:27:35.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:27:35.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149702 -lt 111669149702 2024-09-10T20:27:35.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:27:35.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-10T20:27:35.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:27:35.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:27:35.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-10T20:27:35.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:27:35.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-10T20:27:35.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-10T20:27:35.666 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 137438953474 2024-09-10T20:27:35.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:27:35.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-10T20:27:35.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:27:35.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:35.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:27:36.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:27:36.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:27:36.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:27:36.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:27:36.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:27:36.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:27:36.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:27:36.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:27:36.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:27:36.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:27:36.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:27:36.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:27:36.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:27:36.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:27:36.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:27:36.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:27:36.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:27:36.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:27:36.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:27:37.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:27:37.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:27:37.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:27:37.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:27:37.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:27:37.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:27:37.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:27:37.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:27:37.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:27:37.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:27:37.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:27:37.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:27:37.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:27:37.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:27:37.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:27:37.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:37.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:27:37.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:27:37.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:27:37.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:27:37.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:27:37.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:27:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:27:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:27:37.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:27:37.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:27:37.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:27:37.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:27:37.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:27:37.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:27:37.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:27:37.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:27:37.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:27:37.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:37.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:27:37.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:27:37.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:27:37.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:27:37.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:27:37.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:27:37.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:37.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:27:37.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-09-10T20:27:37.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=testpool 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=testpool 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-10T20:27:37.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-10T20:27:38.747 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 396c73f5-77b4-49be-bd71-2e0f942fb820 2024-09-10T20:27:38.970 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:27:39.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:39.013+0000 7effa2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:39.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:39.016+0000 7effa2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:39.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:39.017+0000 7effa2bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:39.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:39.018+0000 7effa2bcc640 -1 bdev(0x55ed39f1fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:27:39.022 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:39.018+0000 7effa2bcc640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:27:45.291 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:27:45.650 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:27:45.687 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:45.684+0000 7f53f2818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:45.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:45.712+0000 7f53f2818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:45.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:45.714+0000 7f53f2818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:45.926 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:27:46.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:46.804+0000 7f53f2818640 -1 Falling back to public interface 2024-09-10T20:27:47.201 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:27:48.486 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:27:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:27:48.970+0000 7f53f2818640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:27:49.768 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:27:51.112 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:27:52.399 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:27:53.787 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:27:55.074 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:27:56.363 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:27:57.653 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:27:58.967 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:28:00.256 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:28:01.776 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:28:03.302 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:28:04.930 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:28:06.507 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:28:07.868 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:28:09.439 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:28:11.142 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:28:11.427 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42865433,v1:127.0.0.1:6803/42865433] [v2:127.0.0.1:6804/42865433,v1:127.0.0.1:6805/42865433] exists,up 396c73f5-77b4-49be-bd71-2e0f942fb820 2024-09-10T20:28:11.431 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 ec3d98d5-e9ef-4c0c-8201-6a391736c29b 2024-09-10T20:28:11.798 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:28:11.847 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:11.843+0000 7f64c3e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:11.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:11.880+0000 7f64c3e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:11.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:11.881+0000 7f64c3e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:11.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:11.881+0000 7f64c3e15640 -1 bdev(0x564ca1056800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:28:11.885 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:11.881+0000 7f64c3e15640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:28:18.169 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:28:18.533 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:28:18.568 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:18.565+0000 7f8eb261c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:18.570 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:18.567+0000 7f8eb261c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:18.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:18.568+0000 7f8eb261c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:18.810 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:28:19.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:19.936+0000 7f8eb261c640 -1 Falling back to public interface 2024-09-10T20:28:20.105 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:28:21.393 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:28:21.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:21.549+0000 7f8eb261c640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:28:22.764 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:28:24.137 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:28:25.539 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:28:26.874 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:28:28.281 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:28:29.608 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:28:30.926 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:28:32.293 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:28:33.586 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:28:34.874 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:28:36.633 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:28:38.435 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:28:40.176 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:28:41.655 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:28:43.266 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:28:44.554 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:28:45.845 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:28:46.159 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1680659875,v1:127.0.0.1:6811/1680659875] [v2:127.0.0.1:6812/1680659875,v1:127.0.0.1:6813/1680659875] exists,up ec3d98d5-e9ef-4c0c-8201-6a391736c29b 2024-09-10T20:28:46.163 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 69e73625-d481-4ad5-ac59-281586edf775 2024-09-10T20:28:46.543 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:28:46.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:46.590+0000 7f26aa683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:46.625+0000 7f26aa683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:46.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:46.626+0000 7f26aa683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:46.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:46.626+0000 7f26aa683640 -1 bdev(0x564559dbe800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:28:46.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:46.626+0000 7f26aa683640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:28:53.059 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:28:53.417 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:28:53.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:53.449+0000 7f53a6d43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:53.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:53.451+0000 7f53a6d43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:53.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:53.452+0000 7f53a6d43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:53.691 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:28:54.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:54.807+0000 7f53a6d43640 -1 Falling back to public interface 2024-09-10T20:28:54.965 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:28:56.244 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:28:56.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:28:56.745+0000 7f53a6d43640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:28:57.528 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:28:58.851 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:29:00.160 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:29:01.464 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:29:02.798 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:29:04.129 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:29:05.417 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:29:06.708 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:29:07.998 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:29:09.288 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:29:11.081 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:29:13.003 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:29:14.747 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:29:16.675 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:29:18.027 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:29:19.788 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:29:21.243 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:29:21.524 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2516851621,v1:127.0.0.1:6819/2516851621] [v2:127.0.0.1:6820/2516851621,v1:127.0.0.1:6821/2516851621] exists,up 69e73625-d481-4ad5-ac59-281586edf775 2024-09-10T20:29:21.917 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T20:29:23.685 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T20:29:25.249 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672973 2024-09-10T20:29:25.527 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509443 2024-09-10T20:29:28.052 INFO:tasks.workunit.client.0.smithi181.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-09-10T20:29:28.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-09-10T20:29:28.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-09-10T20:29:28.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-09-10T20:29:28.419 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T20:29:28.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-10T20:29:28.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:29:28.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:29:28.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:29:28.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:29:28.431 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:29:28.431 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:29:28.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:29:28.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:29:28.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:29:28.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:29:29.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:29:29.016 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:29:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:29:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:29:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:29.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:29:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T20:29:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T20:29:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T20:29:29.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:29.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:29:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-10T20:29:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-10T20:29:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-09-10T20:29:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:29.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:29:29.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-10T20:29:29.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-10T20:29:29.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509445' 2024-09-10T20:29:29.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:29.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T20:29:29.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:29.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:29:29.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T20:29:29.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:29.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T20:29:29.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T20:29:29.467 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836503 2024-09-10T20:29:29.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:29.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-10T20:29:29.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:29:30.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:29:30.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:31.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T20:29:31.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:31.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-10T20:29:31.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:31.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:29:31.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-10T20:29:31.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:31.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-10T20:29:31.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-10T20:29:31.030 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672975 2024-09-10T20:29:31.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:29:31.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-10T20:29:31.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:31.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-10T20:29:31.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:31.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:29:31.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-10T20:29:31.308 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:31.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-10T20:29:31.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-10T20:29:31.309 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509445 2024-09-10T20:29:31.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:29:31.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-10T20:29:31.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:29:31.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:31.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:29:31.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:29:31.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:29:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:29:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:29:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-09-10T20:29:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:29:32.710 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:29:32.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:32.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:29:32.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:29:32.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:32.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:29:33.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:29:33.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:29:33.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:29:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:29:33.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:29:33.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T20:29:35.314 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T20:29:35.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T20:29:35.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:29:35.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:29:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:29:35.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:29:35.834 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:29:35.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:29:35.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:35.859+0000 7f19a3fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:35.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:35.862+0000 7f19a3fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:35.866 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:35.863+0000 7f19a3fc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:36.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:36.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:37.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:29:37.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:37.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:29:37.401 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:29:37.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:37.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:37.468 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:37.465+0000 7f19a3fc5640 -1 Falling back to public interface 2024-09-10T20:29:37.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:38.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:39.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:39.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:39.276+0000 7f19a3fc5640 -1 osd.0 21 log_to_monitors true 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:40.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:40.302 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1683217379,v1:127.0.0.1:6803/1683217379] [v2:127.0.0.1:6804/1683217379,v1:127.0.0.1:6805/1683217379] exists,up 396c73f5-77b4-49be-bd71-2e0f942fb820 2024-09-10T20:29:40.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:29:40.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:29:40.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:29:40.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:29:40.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:29:40.772 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:29:40.772 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:29:40.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:29:40.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:40.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:29:40.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-10T20:29:40.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-10T20:29:40.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-10T20:29:40.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:40.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:29:41.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-10T20:29:41.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-10T20:29:41.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-09-10T20:29:41.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:41.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:29:41.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T20:29:41.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T20:29:41.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509448' 2024-09-10T20:29:41.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:41.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-10T20:29:41.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:41.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:29:41.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-10T20:29:41.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:41.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-10T20:29:41.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-10T20:29:41.121 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 107374182402 2024-09-10T20:29:41.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:41.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-10T20:29:41.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:29:42.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:29:42.396 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:42.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-10T20:29:42.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:29:43.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:29:43.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:43.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-10T20:29:43.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:43.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-10T20:29:43.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:43.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:29:43.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-10T20:29:43.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:43.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-10T20:29:43.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-10T20:29:43.971 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672979 2024-09-10T20:29:43.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:29:44.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509448 2024-09-10T20:29:44.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:29:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-10T20:29:44.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:29:44.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:44.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:29:44.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:29:44.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:29:45.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:29:45.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:29:45.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:45.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:45.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:29:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:29:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:29:45.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-10T20:29:45.619 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T20:29:45.619 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T20:29:45.620 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:29:45.633 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T20:29:45.633 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-10T20:29:45.633 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:29:45.646 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T20:29:45.646 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-10T20:29:45.646 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:29:45.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-09-10T20:29:45.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T20:29:45.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:29:45.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:45.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:29:45.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-09-10T20:29:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-09-10T20:29:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:29:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:29:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:29:46.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:29:46.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:46.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-09-10T20:29:46.618 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:29:46.618 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:29:46.619 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T20:29:46.619 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-27T20:28:06.615784+0000" 2024-09-10T20:29:46.619 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:29:46.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:29:46.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:29:46.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:29:46.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:29:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:29:21.847001+0000 '>' 2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:29:47.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:29:47.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:29:47.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:29:47.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:29:47.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:29:47.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:29:47.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:29:48.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:29:21.847001+0000 '>' 2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:48.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:29:49.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:29:49.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:29:49.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:29:49.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:29:49.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:29:49.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:29:49.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:29:49.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:29:21.847001+0000 '>' 2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:49.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:29:50.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:29:50.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:29:50.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:29:50.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:29:50.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:29:50.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:29:50.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:29:46.945072+0000 '>' 2024-09-10T20:29:21.847001+0000 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:29:50.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:29:50.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:29:50.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:29:51.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:29:51.137 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:29:51.137 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:29:51.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:29:51.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:51.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:29:51.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182405 2024-09-10T20:29:51.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182405 2024-09-10T20:29:51.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405' 2024-09-10T20:29:51.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:51.241 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:29:51.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T20:29:51.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T20:29:51.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-42949672982' 2024-09-10T20:29:51.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:29:51.344 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:29:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-10T20:29:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-10T20:29:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-42949672982 2-64424509452' 2024-09-10T20:29:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:51.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182405 2024-09-10T20:29:51.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:29:51.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182405 2024-09-10T20:29:51.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182405 2024-09-10T20:29:51.448 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 107374182405 2024-09-10T20:29:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-10T20:29:51.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182404 -lt 107374182405 2024-09-10T20:29:51.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:29:52.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:29:52.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:29:53.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-10T20:29:53.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:53.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T20:29:53.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:53.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:29:53.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T20:29:53.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:53.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T20:29:53.011 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672982 2024-09-10T20:29:53.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T20:29:53.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:29:53.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-10T20:29:53.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:29:53.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-10T20:29:53.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:29:53.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:29:53.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-10T20:29:53.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:29:53.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-10T20:29:53.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-10T20:29:53.303 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509452 2024-09-10T20:29:53.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:29:53.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-10T20:29:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:29:53.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:53.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:53.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:29:53.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:29:53.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:29:54.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:29:54.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:29:54.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:29:54.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:29:54.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:29:54.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:29:54.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:29:54.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-09-10T20:29:54.801 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:29:54.801 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-10T20:29:47.109276+0000 21'1 26:61 [1,0] 1 [1,0] 1 21'1 2024-09-10T20:29:46.945072+0000 21'1 2024-09-10T20:29:46.945072+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:29:46.945072+0000 1 0 2024-09-10T20:29:54.801 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:29:54.801 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:29:54.801 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:29:54.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-10T20:29:54.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:29:54.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:29:54.813 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:29:54.813 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:29:54.814 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:29:54.814 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:54.814 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:29:55.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:29:55.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:29:55.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:29:55.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:29:55.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:29:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:29:55.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:29:55.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:29:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:29:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:29:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:29:55.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:29:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-10T20:29:56.178 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:29:56.179 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:29:56.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:29:56.451 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:29:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:29:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:29:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:29:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:29:56.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:29:56.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:29:56.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:29:56.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:29:56.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:29:56.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:29:56.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:56.477+0000 7f52df214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:56.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:56.479+0000 7f52df214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:56.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:56.480+0000 7f52df214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:29:56.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:29:56.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:29:56.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:29:56.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:56.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:57.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:57.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:57.597+0000 7f52df214640 -1 Falling back to public interface 2024-09-10T20:29:58.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:29:58.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:58.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:29:58.021 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:29:58.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:58.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:29:59.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:29:59.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:29:59.690 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:29:59.687+0000 7f52df214640 -1 osd.0 26 log_to_monitors true 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:00.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1735668849,v1:127.0.0.1:6803/1735668849] [v2:127.0.0.1:6804/1735668849,v1:127.0.0.1:6805/1735668849] exists,up 396c73f5-77b4-49be-bd71-2e0f942fb820 2024-09-10T20:30:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:30:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:30:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:30:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:30:00.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:30:01.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:01.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:30:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-10T20:30:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-10T20:30:01.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-10T20:30:01.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:01.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:30:01.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-10T20:30:01.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-10T20:30:01.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-09-10T20:30:01.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:01.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:30:01.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-10T20:30:01.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-10T20:30:01.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509455' 2024-09-10T20:30:01.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:01.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-10T20:30:01.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:01.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:30:01.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-10T20:30:01.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:01.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-10T20:30:01.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-10T20:30:01.670 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 128849018882 2024-09-10T20:30:01.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:30:01.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-10T20:30:01.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:30:02.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:30:02.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:30:03.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-10T20:30:03.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:03.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-10T20:30:03.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:03.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:30:03.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-10T20:30:03.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:03.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-10T20:30:03.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-10T20:30:03.235 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672985 2024-09-10T20:30:03.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:30:03.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-10T20:30:03.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:03.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-10T20:30:03.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:30:03.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-10T20:30:03.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:03.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-10T20:30:03.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-10T20:30:03.515 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509455 2024-09-10T20:30:03.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:30:03.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-10T20:30:03.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:30:03.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:30:03.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:30:04.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:30:04.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:30:04.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:30:04.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:30:04.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:30:04.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:30:04.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:30:04.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:30:04.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:30:04.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:30:04.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:30:04.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:30:04.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:30:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:30:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:30:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:30:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:30:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:30:05.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:30:05.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:30:05.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:30:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:30:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:30:05.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:30:05.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:30:06.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:30:06.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:30:06.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:30:06.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:30:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:30:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:30:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:30:06.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:30:06.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:06.481+0000 7f2ca69bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:06.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:06.483+0000 7f2ca69bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:06.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:06.484+0000 7f2ca69bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:06.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:30:06.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:30:06.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:30:06.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:07.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:07.845+0000 7f2ca69bf640 -1 Falling back to public interface 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:08.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:08.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:09.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:09.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:09.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:30:09.310 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:30:09.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:09.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:09.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:09.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:09.737+0000 7f2ca69bf640 -1 osd.0 31 log_to_monitors true 2024-09-10T20:30:10.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:10.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:10.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:30:10.590 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:30:10.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:10.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:10.983 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/531500503,v1:127.0.0.1:6803/531500503] [v2:127.0.0.1:6804/531500503,v1:127.0.0.1:6805/531500503] exists,up 396c73f5-77b4-49be-bd71-2e0f942fb820 2024-09-10T20:30:10.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:30:10.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:30:10.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:30:11.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:30:11.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:30:11.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:30:11.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:30:11.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:30:11.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:11.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:30:11.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-10T20:30:11.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-10T20:30:11.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-10T20:30:11.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:11.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:30:11.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-10T20:30:11.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-10T20:30:11.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672988' 2024-09-10T20:30:11.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:30:11.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672988 2-64424509458' 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-10T20:30:11.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:11.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:30:11.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-10T20:30:11.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:11.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-10T20:30:11.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-10T20:30:11.707 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 150323855362 2024-09-10T20:30:11.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:30:11.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 150323855362 2024-09-10T20:30:11.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:30:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:30:12.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:30:13.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-10T20:30:13.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:13.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-10T20:30:13.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:13.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:30:13.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-10T20:30:13.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:13.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-10T20:30:13.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-10T20:30:13.268 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672988 2024-09-10T20:30:13.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:30:13.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-10T20:30:13.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:30:13.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T20:30:13.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:30:13.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:30:13.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T20:30:13.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:30:13.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T20:30:13.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T20:30:13.547 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509458 2024-09-10T20:30:13.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:30:13.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-10T20:30:13.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:30:13.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:30:13.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:30:14.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:30:14.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:30:14.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:30:14.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:30:14.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:30:14.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:30:14.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:30:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:30:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:30:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:30:14.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:30:14.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:30:14.789 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:29:46.944+0000 7f8e96f86640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:30:14.789 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:29:46.944+0000 7f8e96f86640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:30:14.789 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:29:46.944+0000 7f8e96f86640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-10T20:30:14.789 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:29:46.944+0000 7f8e96f86640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-10T20:30:14.789 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:29:46.944+0000 7f8e96f86640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:30:14.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:30:14.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:30:14.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:30:14.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:30:14.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:30:14.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:30:14.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:30:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:30:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:30:14.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:30:14.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:14.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:30:14.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:30:14.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:14.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:30:14.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:30:14.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:30:14.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:30:14.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:14.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:14.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:30:14.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:30:14.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:30:14.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:30:14.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:30:14.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:30:14.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:30:14.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:30:14.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:30:14.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:30:14.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:14.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:30:14.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:30:14.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:30:14.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:30:14.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:30:14.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:30:14.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:30:14.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:14.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:14.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:30:14.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:30:14.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:30:14.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:30:14.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:30:14.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:14.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:14.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:30:14.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:30:14.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:30:14.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:30:14.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:30:14.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:30:15.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:30:15.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:15.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:15.385 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:15.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:30:15.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:30:15.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:30:15.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:30:15.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:30:15.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:30:15.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:15.565 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:30:15.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:30:15.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:30:15.643 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:15.644 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:15.644 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:30:15.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:30:15.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:30:15.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:30:15.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:30:16.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:30:16.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:16.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:16.537 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:16.538 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:16.538 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:16.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:16.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:30:16.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:30:16.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:30:16.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-09-10T20:30:16.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-10T20:30:16.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:30:16.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:30:16.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:30:16.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:30:16.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:30:16.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b77a355f-0007-4daf-ad24-ea521b249b7f' 2024-09-10T20:30:16.573 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:30:16.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:30:16.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBYrOBmQa3jIhAAUTZ0R4/zdBWE3CJH4VkL0w== 2024-09-10T20:30:16.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBYrOBmQa3jIhAAUTZ0R4/zdBWE3CJH4VkL0w=="}' 2024-09-10T20:30:16.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b77a355f-0007-4daf-ad24-ea521b249b7f -i td/osd-scrub-repair/0/new.json 2024-09-10T20:30:16.793 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:30:16.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:30:16.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBYrOBmQa3jIhAAUTZ0R4/zdBWE3CJH4VkL0w== --osd-uuid b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:30:16.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:16.833+0000 7f0ce9bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:16.841+0000 7f0ce9bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:16.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:16.842+0000 7f0ce9bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:16.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:16.842+0000 7f0ce9bcc640 -1 bdev(0x555d0a733c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:30:16.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:16.842+0000 7f0ce9bcc640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:30:22.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:30:22.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:30:22.345 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:30:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:30:22.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:30:22.727 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:30:22.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:30:22.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:30:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:30:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:30:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:30:22.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:22.754+0000 7f7d24345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:22.759 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:22.756+0000 7f7d24345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:22.757+0000 7f7d24345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:23.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:30:23.003 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:30:23.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:23.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:23.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:24.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:24.117+0000 7f7d24345640 -1 Falling back to public interface 2024-09-10T20:30:24.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:24.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:24.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:30:24.279 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:30:24.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:24.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:24.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:25.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:25.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:25.570 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:30:25.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:30:25.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:25.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:25.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:26.315 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:26.312+0000 7f7d24345640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:30:26.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:26.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:26.860 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:30:26.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:30:26.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:26.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:27.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:28.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:28.196 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:30:28.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:28.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:30:28.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:28.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:28.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:29.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:29.482 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:30:29.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:29.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:30:29.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:29.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:29.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:30.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:30.769 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:30:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:30:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:31.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:32.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:32.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:32.097 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:30:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:30:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:32.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:33.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:33.383 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:30:33.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:33.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:30:33.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:33.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:33.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:34.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:34.675 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:30:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:30:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:35.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:36.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:36.005 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:30:36.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:36.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:30:36.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:36.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:36.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:37.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:37.292 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:30:37.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:30:37.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:37.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:37.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:38.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:38.578 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:30:38.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:38.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:30:38.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:38.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:39.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:40.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:40.121 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:30:40.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:40.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:30:40.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:40.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:40.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:41.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:41.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:41.593 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:30:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:30:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:41.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:42.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:43.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:30:43.406 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:30:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:43.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:43.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:44.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:44.704 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:30:44.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:44.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:30:44.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:44.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:45.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:46.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:46.374 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:30:46.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:46.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:30:46.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:46.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:46.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:47.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:47.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:47.851 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:30:47.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:30:47.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:47.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:48.172 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1242782516,v1:127.0.0.1:6803/1242782516] [v2:127.0.0.1:6804/1242782516,v1:127.0.0.1:6805/1242782516] exists,up b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:30:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:30:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:30:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:30:48.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:30:48.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1f050565-7060-4e55-9d3a-db4a23a560fc' 2024-09-10T20:30:48.179 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:30:48.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:30:48.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4rOBmxaFZCxAAzsZGa88dmdLzie3NBx1YvA== 2024-09-10T20:30:48.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4rOBmxaFZCxAAzsZGa88dmdLzie3NBx1YvA=="}' 2024-09-10T20:30:48.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f050565-7060-4e55-9d3a-db4a23a560fc -i td/osd-scrub-repair/1/new.json 2024-09-10T20:30:48.518 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:30:48.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:30:48.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB4rOBmxaFZCxAAzsZGa88dmdLzie3NBx1YvA== --osd-uuid 1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:30:48.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:48.558+0000 7f11aaa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:48.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:48.599+0000 7f11aaa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:48.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:48.600+0000 7f11aaa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:48.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:48.600+0000 7f11aaa2c640 -1 bdev(0x55d8574f9c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:30:48.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:48.600+0000 7f11aaa2c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:30:54.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:30:54.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:30:54.571 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:30:54.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:30:54.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:30:54.931 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:30:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:30:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:30:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:30:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:30:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:30:54.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:54.957+0000 7fc7ca440640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:54.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:54.959+0000 7fc7ca440640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:54.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:54.960+0000 7fc7ca440640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:30:55.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:55.210 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:30:55.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:30:55.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:55.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:30:55.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:56.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:56.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:56.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:30:56.484 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:30:56.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:56.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:30:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:56.593+0000 7fc7ca440640 -1 Falling back to public interface 2024-09-10T20:30:56.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:57.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:57.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:57.790 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:30:57.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:30:57.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:57.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:30:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:30:58.496+0000 7fc7ca440640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:30:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:59.083 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:30:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:30:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:30:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:00.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:00.371 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:00.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:00.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:01.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:01.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:01.663 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:31:01.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:31:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:01.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:02.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:02.951 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:31:02.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:02.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:31:02.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:02.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:03.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:04.242 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:31:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:31:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:04.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:05.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:05.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:05.532 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:31:05.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:31:05.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:05.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:05.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:06.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:06.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:06.819 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:31:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:31:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:07.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:08.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:08.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:08.213 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:31:08.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:31:08.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:08.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:08.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:09.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:09.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:09.617 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:31:09.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:31:09.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:09.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:09.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:10.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:10.905 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:31:10.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:10.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:31:10.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:10.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:11.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:12.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:12.344 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:31:12.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:12.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:31:12.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:12.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:12.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:13.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:13.729 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:31:13.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:13.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:31:13.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:13.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:14.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:15.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:15.465 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:31:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:31:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:15.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:15.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:16.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:16.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:16.866 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:31:16.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:31:16.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:16.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:17.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:18.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:18.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:18.629 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:31:18.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:31:18.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:18.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:20.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:20.197 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:31:20.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:20.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:31:20.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:20.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:20.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:21.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:21.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:21.839 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:31:21.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:31:21.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:21.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:22.126 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1459587242,v1:127.0.0.1:6811/1459587242] [v2:127.0.0.1:6812/1459587242,v1:127.0.0.1:6813/1459587242] exists,up 1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:31:22.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:31:22.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:31:22.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:31:22.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:31:22.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8726f2e9-09c8-4e7d-90be-99e8838bf20e 2024-09-10T20:31:22.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8726f2e9-09c8-4e7d-90be-99e8838bf20e' 2024-09-10T20:31:22.134 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 8726f2e9-09c8-4e7d-90be-99e8838bf20e 2024-09-10T20:31:22.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:31:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCarOBm7h+mCBAAcW0qLfnijDnzF6vbDW+E6A== 2024-09-10T20:31:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCarOBm7h+mCBAAcW0qLfnijDnzF6vbDW+E6A=="}' 2024-09-10T20:31:22.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8726f2e9-09c8-4e7d-90be-99e8838bf20e -i td/osd-scrub-repair/2/new.json 2024-09-10T20:31:22.583 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:31:22.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:31:22.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCarOBm7h+mCBAAcW0qLfnijDnzF6vbDW+E6A== --osd-uuid 8726f2e9-09c8-4e7d-90be-99e8838bf20e 2024-09-10T20:31:22.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:22.624+0000 7f9c8ec0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:22.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:22.673+0000 7f9c8ec0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:22.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:22.674+0000 7f9c8ec0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:22.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:22.674+0000 7f9c8ec0d640 -1 bdev(0x559a12672800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:31:22.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:22.674+0000 7f9c8ec0d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:31:28.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:31:28.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:31:28.611 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:31:28.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:31:28.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:31:28.990 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:31:28.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:31:28.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:31:28.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:31:28.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:31:28.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:31:29.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:29.016+0000 7f90bb6eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:29.021 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:29.018+0000 7f90bb6eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:29.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:29.019+0000 7f90bb6eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:29.267 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:31:29.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:31:29.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:29.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:29.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:30.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:30.143+0000 7f90bb6eb640 -1 Falling back to public interface 2024-09-10T20:31:30.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:30.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:30.543 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:31:30.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:31:30.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:30.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:31.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:31.834 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:31:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:31:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:31.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:31.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:31:31.970+0000 7f90bb6eb640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:31:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:33.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:33.209 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:31:33.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:33.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:31:33.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:33.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:33.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:34.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:34.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:34.599 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:31:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:31:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:35.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:36.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:36.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:36.003 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:31:36.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:31:36.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:36.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:36.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:37.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:37.291 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:31:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:31:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:37.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:38.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:38.602 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:31:38.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:38.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:31:38.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:38.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:38.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:39.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:39.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:39.904 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:31:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:31:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:40.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:41.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:41.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:41.187 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:31:41.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:31:41.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:41.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:41.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:42.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:42.476 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:31:42.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:42.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:31:42.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:42.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:42.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:43.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:43.874 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:31:43.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:43.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:31:43.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:43.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:44.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:45.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:45.354 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:31:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:31:45.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:45.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:46.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:46.963 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:31:46.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:46.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:31:46.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:46.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:47.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:48.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:48.647 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:31:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:31:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:48.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:49.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:50.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:50.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:50.015 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:31:50.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:31:50.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:50.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:50.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:51.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:51.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:51.461 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:31:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:31:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:52.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:53.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:53.056 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:31:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:31:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:53.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:54.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:54.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:54.765 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:31:54.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:31:54.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:54.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:55.075 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/531296772,v1:127.0.0.1:6819/531296772] [v2:127.0.0.1:6820/531296772,v1:127.0.0.1:6821/531296772] exists,up 8726f2e9-09c8-4e7d-90be-99e8838bf20e 2024-09-10T20:31:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-09-10T20:31:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-10T20:31:55.466 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T20:31:55.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:31:56.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-09-10T20:31:57.004 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T20:31:57.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-10T20:31:57.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:31:57.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:31:57.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:31:57.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:31:57.018 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:31:57.018 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:31:57.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:31:57.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:31:57.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:31:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:31:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:31:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:31:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:31:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:31:57.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:31:57.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:31:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:31:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:31:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:31:57.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:57.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:31:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T20:31:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T20:31:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T20:31:57.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:57.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:31:57.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-10T20:31:57.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-10T20:31:57.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672973' 2024-09-10T20:31:57.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:57.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672973 2-64424509447' 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T20:31:57.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:57.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:31:57.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T20:31:57.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:57.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T20:31:57.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T20:31:57.834 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T20:31:57.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:31:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T20:31:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:31:59.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:31:59.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:31:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-10T20:31:59.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:59.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-10T20:31:59.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:59.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:31:59.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-10T20:31:59.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:59.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-10T20:31:59.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-10T20:31:59.400 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672973 2024-09-10T20:31:59.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:31:59.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672973 2024-09-10T20:31:59.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:59.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-10T20:31:59.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:59.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:31:59.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-10T20:31:59.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:59.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-10T20:31:59.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-10T20:31:59.681 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509447 2024-09-10T20:31:59.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:31:59.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-10T20:31:59.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:31:59.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:31:59.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:00.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:00.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:00.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:00.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:00.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:00.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:00.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:00.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:32:00.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:32:00.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:32:00.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:00.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:32:01.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:01.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:01.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:01.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:01.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:32:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:32:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:32:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:32:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:01.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:02.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:02.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:02.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:02.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:02.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:02.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:02.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:02.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:32:02.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:32:03.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:32:03.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:32:03.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T20:32:03.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:03.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:32:03.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:03.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:03.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:03.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:03.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:03.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:04.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:32:04.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:32:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:32:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:32:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 13 )) 2024-09-10T20:32:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:04.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:32:05.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:05.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:05.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:05.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:05.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:05.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:05.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:05.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:05.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:32:05.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:05.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:05.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:32:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:32:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:32:05.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-09-10T20:32:05.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-09-10T20:32:05.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-09-10T20:32:05.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:05.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:06.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:06.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:07.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:07.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:07.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:07.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:07.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-10T20:32:07.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:32:07.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:32:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:32:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:32:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:32:07.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:32:07.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:32:07.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:32:07.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:32:07.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:32:07.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:08.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-10T20:32:10.049 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:10.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:10.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:32:10.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:32:10.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:32:10.573 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:32:10.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:10.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:32:10.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:32:10.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:10.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:10.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:10.604 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:10.600+0000 7f1cdde24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:10.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:10.603+0000 7f1cdde24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:10.607 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:10.604+0000 7f1cdde24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:10.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:10.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:11.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:11.735 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:11.732+0000 7f1cdde24640 -1 Falling back to public interface 2024-09-10T20:32:12.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:12.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:12.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:12.133 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:12.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:12.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:12.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:13.314 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:13.311+0000 7f1cdde24640 -1 osd.0 21 log_to_monitors true 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1504446125,v1:127.0.0.1:6803/1504446125] [v2:127.0.0.1:6804/1504446125,v1:127.0.0.1:6805/1504446125] exists,up b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:32:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:32:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:32:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:32:13.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:32:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:32:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:32:13.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:32:13.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:32:13.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:32:13.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:32:14.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:32:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:32:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:32:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:14.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:32:14.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-10T20:32:14.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-10T20:32:14.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-10T20:32:14.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:14.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:32:14.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-10T20:32:14.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-10T20:32:14.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-09-10T20:32:14.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:14.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:32:14.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T20:32:14.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T20:32:14.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509451' 2024-09-10T20:32:14.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:14.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-10T20:32:14.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:14.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:32:14.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-10T20:32:14.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-10T20:32:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-10T20:32:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 107374182402 2024-09-10T20:32:14.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 107374182402 2024-09-10T20:32:14.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:32:15.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:32:15.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:15.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-10T20:32:15.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:15.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-10T20:32:15.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:15.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:32:15.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-10T20:32:15.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:15.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-10T20:32:15.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-10T20:32:15.997 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672978 2024-09-10T20:32:15.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:32:16.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-10T20:32:16.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:16.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T20:32:16.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:16.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:32:16.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T20:32:16.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:16.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T20:32:16.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T20:32:16.275 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509451 2024-09-10T20:32:16.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:32:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-10T20:32:16.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:32:16.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:16.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:16.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:16.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:17.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:17.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:17.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:17.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:17.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:17.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:32:17.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:32:17.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:32:17.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:17.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:32:17.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:17.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:17.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:17.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:17.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:17.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:17.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:17.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:17.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:32:17.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:17.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:17.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:18.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:32:18.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:32:18.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:32:18.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:32:18.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:32:18.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:32:18.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:32:18.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:18.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:19.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:32:19.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:19.069 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:19.069 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:32:19.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:32:19.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:32:19.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-10T20:32:19.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:32:19.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:32:19.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:32:19.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:32:19.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:32:19.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:32:19.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:32:19.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:32:19.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:32:19.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:32:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:32:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:20.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-10T20:32:21.725 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:104778fc:::obj2:head# 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:32:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:32:22.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:32:22.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:22.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:22.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:22.277+0000 7fed25e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:22.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:22.279+0000 7fed25e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:22.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:22.281+0000 7fed25e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:22.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:22.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:22.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:23.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:23.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:23.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:23.856 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:23.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:23.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:23.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:23.877+0000 7fed25e31640 -1 Falling back to public interface 2024-09-10T20:32:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:25.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:25.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:25.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:25.137 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:32:25.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:25.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:25.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:25.442+0000 7fed25e31640 -1 osd.0 26 log_to_monitors true 2024-09-10T20:32:26.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:26.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:26.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:32:26.425 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:32:26.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:26.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3644743715,v1:127.0.0.1:6803/3644743715] [v2:127.0.0.1:6804/3644743715,v1:127.0.0.1:6805/3644743715] exists,up b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:32:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:32:26.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:32:26.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:32:26.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:32:26.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:32:26.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:32:26.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:32:26.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:27.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:32:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-10T20:32:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-10T20:32:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-10T20:32:27.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:27.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:32:27.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-10T20:32:27.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-10T20:32:27.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672981' 2024-09-10T20:32:27.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:27.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:32:27.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-10T20:32:27.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-10T20:32:27.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672981 2-64424509455' 2024-09-10T20:32:27.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:27.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-10T20:32:27.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:27.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:32:27.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-10T20:32:27.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:27.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-10T20:32:27.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-10T20:32:27.493 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 128849018882 2024-09-10T20:32:27.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:27.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-10T20:32:27.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:32:28.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:32:28.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:29.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-10T20:32:29.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:29.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-10T20:32:29.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:29.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:32:29.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-10T20:32:29.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:29.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-10T20:32:29.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-10T20:32:29.049 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672981 2024-09-10T20:32:29.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:32:29.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672981 2024-09-10T20:32:29.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:29.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-10T20:32:29.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:29.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:32:29.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-10T20:32:29.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:29.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-10T20:32:29.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-10T20:32:29.326 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509455 2024-09-10T20:32:29.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:32:29.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-10T20:32:29.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:32:29.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:29.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:29.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:29.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:30.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:32:30.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:30.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:30.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:32:30.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:32:30.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:32:30.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:32:30.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:32:30.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:32:30.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:32:30.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:32:31.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:32:31.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:32:31.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:32:31.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:32:31.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:32:31.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:32:31.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:32:31.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-10T20:32:33.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:33.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:32:33.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:32:33.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:33.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:33.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:32:33.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:33.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:33.192+0000 7fcc568f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:33.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:33.195+0000 7fcc568f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:33.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:33.196+0000 7fcc568f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:33.449 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:32:33.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:33.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:33.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:34.557 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:34.552+0000 7fcc568f1640 -1 Falling back to public interface 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:34.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:36.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:36.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:36.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:36.011 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:32:36.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:36.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:36.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:36.129+0000 7fcc568f1640 -1 osd.1 31 log_to_monitors true 2024-09-10T20:32:36.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:37.590 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/4167486581,v1:127.0.0.1:6811/4167486581] [v2:127.0.0.1:6812/4167486581,v1:127.0.0.1:6813/4167486581] exists,up 1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:32:37.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:32:37.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:32:37.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:32:37.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:32:37.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:32:37.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:32:37.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:32:37.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:37.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:32:38.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-10T20:32:38.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-10T20:32:38.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885' 2024-09-10T20:32:38.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:38.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:32:38.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-10T20:32:38.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-10T20:32:38.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-09-10T20:32:38.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:38.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509458' 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018885 2024-09-10T20:32:38.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:38.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:32:38.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018885 2024-09-10T20:32:38.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:38.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-10T20:32:38.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-09-10T20:32:38.266 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 128849018885 2024-09-10T20:32:38.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:38.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018884 -lt 128849018885 2024-09-10T20:32:38.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:32:39.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:32:39.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:39.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018885 -lt 128849018885 2024-09-10T20:32:39.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:39.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855362 2024-09-10T20:32:39.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:39.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:32:39.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855362 2024-09-10T20:32:39.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:39.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-10T20:32:39.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-09-10T20:32:39.834 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 150323855362 2024-09-10T20:32:39.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:32:40.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-10T20:32:40.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:40.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T20:32:40.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:40.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:32:40.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T20:32:40.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:40.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T20:32:40.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T20:32:40.111 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509458 2024-09-10T20:32:40.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:32:40.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-10T20:32:40.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:32:40.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:40.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:40.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:41.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:32:41.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:41.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:41.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:41.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-10T20:32:41.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:32:41.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-09-10T20:32:41.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-09-10T20:32:41.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:32:41.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T20:32:41.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T20:32:41.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:32:41.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-09-10T20:32:41.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-09-10T20:32:41.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:41.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:41.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:41.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:42.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:42.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-09-10T20:32:42.299 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:32:42.299 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:32:42.299 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T20:32:42.299 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-27T20:31:02.295919+0000" 2024-09-10T20:32:42.299 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:32:42.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:42.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T20:32:42.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:42.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:32:42.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:42.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:42.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:31:55.414900+0000 '>' 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:42.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:32:43.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:32:43.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:32:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:32:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:43.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:31:55.414900+0000 '>' 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:43.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:32:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:32:44.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:32:44.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:32:44.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:44.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:44.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:44.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:45.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:31:55.414900+0000 '>' 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:45.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:32:46.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:32:46.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:32:46.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:32:46.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:46.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:46.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:46.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:46.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:31:55.414900+0000 '>' 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:46.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:32:47.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:32:47.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:32:47.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:32:47.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:32:47.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:32:47.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:32:47.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:32:44.305710+0000 '>' 2024-09-10T20:31:55.414900+0000 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:32:47.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:32:47.715 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:32:47.715 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:32:47.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:32:47.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:32:47.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:32:47.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:32:47.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:32:47.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:32:47.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:32:47.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:32:47.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:32:48.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:32:48.079 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:48.079 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:32:48.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:32:48.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:48.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:32:48.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018888 2024-09-10T20:32:48.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018888 2024-09-10T20:32:48.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888' 2024-09-10T20:32:48.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:48.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:32:48.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-10T20:32:48.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-10T20:32:48.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-09-10T20:32:48.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:48.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:32:48.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-10T20:32:48.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-10T20:32:48.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509461' 2024-09-10T20:32:48.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:48.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018888 2024-09-10T20:32:48.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:48.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:32:48.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018888 2024-09-10T20:32:48.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:48.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018888 2024-09-10T20:32:48.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-09-10T20:32:48.391 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 128849018888 2024-09-10T20:32:48.391 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:48.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-10T20:32:48.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:48.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855365 2024-09-10T20:32:48.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:48.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:32:48.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855365 2024-09-10T20:32:48.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:48.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-10T20:32:48.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-09-10T20:32:48.672 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 150323855365 2024-09-10T20:32:48.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:32:48.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-10T20:32:48.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:48.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-10T20:32:48.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:48.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:32:48.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-10T20:32:48.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:48.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-10T20:32:48.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-10T20:32:48.980 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509461 2024-09-10T20:32:48.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:32:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-10T20:32:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:32:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:49.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:32:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:32:49.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:32:49.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:32:49.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:32:49.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:32:50.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:32:50.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:32:50.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:32:50.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-10T20:32:50.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:32:50.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:32:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:32:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:32:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:32:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:50.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:32:50.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-10T20:32:50.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-10T20:32:50.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890' 2024-09-10T20:32:50.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:50.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:32:50.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855366 2024-09-10T20:32:50.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855366 2024-09-10T20:32:50.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855366' 2024-09-10T20:32:50.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:32:50.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:32:50.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-10T20:32:50.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-10T20:32:50.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855366 2-64424509463' 2024-09-10T20:32:50.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:50.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018890 2024-09-10T20:32:50.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:50.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:32:50.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018890 2024-09-10T20:32:50.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:50.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-10T20:32:50.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-09-10T20:32:50.926 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 128849018890 2024-09-10T20:32:50.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:51.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018888 -lt 128849018890 2024-09-10T20:32:51.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:32:52.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:32:52.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:52.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018888 -lt 128849018890 2024-09-10T20:32:52.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:32:53.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:32:53.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:32:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018890 -lt 128849018890 2024-09-10T20:32:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:53.778 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855366 2024-09-10T20:32:53.778 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:53.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:32:53.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855366 2024-09-10T20:32:53.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:53.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855366 2024-09-10T20:32:53.781 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 150323855366 2024-09-10T20:32:53.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855366' 2024-09-10T20:32:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:32:54.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855366 2024-09-10T20:32:54.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:32:54.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-10T20:32:54.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:32:54.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:32:54.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-10T20:32:54.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:32:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-10T20:32:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-10T20:32:54.059 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509463 2024-09-10T20:32:54.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:32:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-10T20:32:54.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:32:54.342 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:32:54.343 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-10T20:32:54.343 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-10T20:32:54.343 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:32:54.343 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-10T20:32:54.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-09-10T20:32:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-10T20:32:54.605 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:32:54.605 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-10T20:32:44.305785+0000 21'10 36:102 [1,0] 1 [1,0] 1 21'10 2024-09-10T20:32:44.305710+0000 21'10 2024-09-10T20:32:44.305710+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:32:44.305710+0000 10 0 2024-09-10T20:32:54.605 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:32:54.605 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:32:54.605 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:32:54.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-10T20:32:54.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-09-10T20:32:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T20:32:54.898 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:32:55.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:32:55.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-10T20:32:55.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-09-10T20:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:32:55.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:32:55.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:32:55.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:32:55.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:55.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-09-10T20:32:56.242 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:32:56.242 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:32:56.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:56.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:32:56.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:56.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:56.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:56.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:32:56.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:32:56.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:32:56.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:56.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:56.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:56.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:56.538+0000 7f965e5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:56.570+0000 7f965e5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:56.571+0000 7f965e5b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:56.794 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:32:56.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:56.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:57.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:57.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:57.943+0000 7f965e5b2640 -1 Falling back to public interface 2024-09-10T20:32:58.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:32:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:58.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:59.358 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:32:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:59.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:59.852 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:32:59.848+0000 7f965e5b2640 -1 osd.0 36 log_to_monitors true 2024-09-10T20:33:00.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:00.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:00.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:00.641 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:33:00.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:00.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/100597856,v1:127.0.0.1:6803/100597856] [v2:127.0.0.1:6804/100597856,v1:127.0.0.1:6805/100597856] exists,up b77a355f-0007-4daf-ad24-ea521b249b7f 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:33:00.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:33:00.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:33:00.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:33:01.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:01.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:33:01.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-10T20:33:01.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-10T20:33:01.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842' 2024-09-10T20:33:01.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:01.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:33:01.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-10T20:33:01.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-10T20:33:01.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-09-10T20:33:01.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:01.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:33:01.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-10T20:33:01.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-10T20:33:01.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509466' 2024-09-10T20:33:01.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:01.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691842 2024-09-10T20:33:01.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:01.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:33:01.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691842 2024-09-10T20:33:01.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-10T20:33:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-09-10T20:33:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 171798691842 2024-09-10T20:33:01.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:01.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 171798691842 2024-09-10T20:33:01.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:33:02.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:33:02.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:03.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-10T20:33:03.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855370 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855370 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:03.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-10T20:33:03.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-09-10T20:33:03.183 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 150323855370 2024-09-10T20:33:03.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:33:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-10T20:33:03.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:03.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-10T20:33:03.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:03.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:33:03.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-10T20:33:03.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-10T20:33:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-10T20:33:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509466 2024-09-10T20:33:03.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:33:03.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-10T20:33:03.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:33:03.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:03.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:04.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:33:04.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:33:04.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:33:04.395 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:33:04.395 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:04.395 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:04.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:33:04.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:33:04.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:33:04.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:33:04.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:33:04.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:33:04.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:33:05.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:33:05.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:33:05.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:05.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:05.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:33:05.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:33:05.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:33:05.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:33:05.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:33:05.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:33:05.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:33:05.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:33:06.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:33:06.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:33:06.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:33:06.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:06.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:06.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:33:06.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:33:06.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:33:06.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:33:06.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T20:33:06.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:33:06.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:33:07.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:33:07.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:33:07.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:33:07.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:33:07.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:33:07.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:33:07.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:33:07.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:33:07.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:33:07.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:33:07.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:07.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:08.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:33:08.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:33:08.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:33:08.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-09-10T20:33:08.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:33:08.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:33:08.034 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:33:08.035 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-10T20:33:08.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-10T20:33:08.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:33:08.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:33:08.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:33:08.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-09-10T20:33:08.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:33:08.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:33:08.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:33:08.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-09-10T20:33:08.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:33:08.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:33:08.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:33:08.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:33:08.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:33:08.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-09-10T20:33:09.906 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-09-10T20:33:09.906 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:104778fc:::obj2:head# 2024-09-10T20:33:10.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:33:10.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:33:10.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:33:10.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:10.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:33:10.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:33:10.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:33:10.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:33:10.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:33:10.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:33:10.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:33:10.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:10.453+0000 7f5bb542f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:10.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:10.456+0000 7f5bb542f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:10.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:10.457+0000 7f5bb542f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:33:10.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:10.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:33:10.729 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:33:10.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:10.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:11.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:11.583+0000 7f5bb542f640 -1 Falling back to public interface 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:12.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:13.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:13.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:13.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:33:13.337 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:33:13.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:13.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:13.385+0000 7f5bb542f640 -1 osd.1 41 log_to_monitors true 2024-09-10T20:33:13.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:14.956 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/1405542825,v1:127.0.0.1:6811/1405542825] [v2:127.0.0.1:6812/1405542825,v1:127.0.0.1:6813/1405542825] exists,up 1f050565-7060-4e55-9d3a-db4a23a560fc 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:33:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:33:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:33:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:33:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:33:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:33:15.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:33:15.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:33:15.320 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:33:15.320 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:33:15.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:33:15.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:15.321 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:33:15.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691846 2024-09-10T20:33:15.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691846 2024-09-10T20:33:15.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846' 2024-09-10T20:33:15.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:15.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:33:15.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528322 2024-09-10T20:33:15.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528322 2024-09-10T20:33:15.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846 1-193273528322' 2024-09-10T20:33:15.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:15.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:33:15.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509470 2024-09-10T20:33:15.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509470 2024-09-10T20:33:15.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846 1-193273528322 2-64424509470' 2024-09-10T20:33:15.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:15.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691846 2024-09-10T20:33:15.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:15.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:33:15.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691846 2024-09-10T20:33:15.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:15.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691846 2024-09-10T20:33:15.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691846' 2024-09-10T20:33:15.632 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 171798691846 2024-09-10T20:33:15.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:15.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691846 2024-09-10T20:33:15.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:33:16.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:33:16.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:17.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691846 -lt 171798691846 2024-09-10T20:33:17.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:17.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528322 2024-09-10T20:33:17.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:17.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:33:17.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528322 2024-09-10T20:33:17.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:17.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528322 2024-09-10T20:33:17.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-09-10T20:33:17.196 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 193273528322 2024-09-10T20:33:17.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:33:17.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528322 -lt 193273528322 2024-09-10T20:33:17.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:17.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509470 2024-09-10T20:33:17.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509470 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509470 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-10T20:33:17.479 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509470 2024-09-10T20:33:17.480 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:33:17.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-10T20:33:17.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:33:17.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:17.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:33:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:33:18.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:33:18.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:33:18.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:33:18.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:18.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:19.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:19.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T20:33:19.275 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:19.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:19.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:32:44.305710+0000 '>' 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:33:20.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:33:20.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:33:20.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:33:20.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:20.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:20.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:20.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:20.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:32:44.305710+0000 '>' 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:20.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:33:21.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:33:21.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:33:21.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:33:21.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:21.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:21.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:21.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:22.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:32:44.305710+0000 '>' 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:22.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:33:23.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:33:23.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:33:23.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:33:23.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:23.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:23.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:23.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:23.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:32:44.305710+0000 '>' 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:23.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:33:24.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:33:24.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:33:24.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:33:24.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:33:24.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:33:24.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:33:24.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:33:24.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:33:19.457745+0000 '>' 2024-09-10T20:32:44.305710+0000 2024-09-10T20:33:24.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:33:24.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:33:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-09-10T20:33:28.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-10T20:33:28.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:33:28.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:33:28.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:33:28.980 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:33:28.981 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:33:28.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:33:28.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:28.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:33:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691849 2024-09-10T20:33:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691849 2024-09-10T20:33:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849' 2024-09-10T20:33:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:33:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528326 2024-09-10T20:33:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528326 2024-09-10T20:33:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849 1-193273528326' 2024-09-10T20:33:29.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:33:29.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:33:29.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509473 2024-09-10T20:33:29.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509473 2024-09-10T20:33:29.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849 1-193273528326 2-64424509473' 2024-09-10T20:33:29.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:29.331 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691849 2024-09-10T20:33:29.331 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:29.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:33:29.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691849 2024-09-10T20:33:29.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:29.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691849 2024-09-10T20:33:29.333 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 171798691849 2024-09-10T20:33:29.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-09-10T20:33:29.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:29.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691848 -lt 171798691849 2024-09-10T20:33:29.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:33:30.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:33:30.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:33:30.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691850 -lt 171798691849 2024-09-10T20:33:30.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:30.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528326 2024-09-10T20:33:30.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:30.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:33:30.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528326 2024-09-10T20:33:30.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:30.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528326 2024-09-10T20:33:30.935 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 193273528326 2024-09-10T20:33:30.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528326' 2024-09-10T20:33:30.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:33:31.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528326 -lt 193273528326 2024-09-10T20:33:31.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:33:31.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509473 2024-09-10T20:33:31.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:33:31.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:33:31.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509473 2024-09-10T20:33:31.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:33:31.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509473 2024-09-10T20:33:31.213 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509473 2024-09-10T20:33:31.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-10T20:33:31.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:33:31.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509473 2024-09-10T20:33:31.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-10T20:33:31.748 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:33:31.748 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-09-10T20:33:19.457847+0000 21'10 46:141 [1,0] 1 [1,0] 1 21'10 2024-09-10T20:33:19.457745+0000 21'10 2024-09-10T20:33:19.457745+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:33:19.457745+0000 9 0 2024-09-10T20:33:31.748 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:33:31.748 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:33:31.748 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:33:31.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-10T20:33:31.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-09-10T20:33:32.018 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:33:32.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:33:32.031 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:33:32.031 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:33:32.031 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-10T20:33:32.031 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:43.269+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:32:44.303+0000 7fcc3c867640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-10T20:33:32.032 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:33:19.456+0000 7f5b9a96c640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-09-10T20:33:32.033 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:33:19.456+0000 7f5b9a96c640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:33:32.033 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:33:19.456+0000 7f5b9a96c640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-10T20:33:32.033 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:33:19.456+0000 7f5b9a96c640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-09-10T20:33:32.033 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:33:19.456+0000 7f5b9a96c640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:33:32.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:33:32.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:33:32.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:33:32.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:33:32.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:33:32.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:33:32.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:33:32.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:33:32.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:33:32.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:33:32.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:33:32.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:33:32.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:33:32.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:33:32.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:33:32.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:33:32.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:33:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:33:32.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:33:32.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:33:32.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:33:32.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:33:32.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:33:32.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:33:32.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:33:32.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:33:32.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:33:32.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:33:32.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:33:32.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:33:32.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:33:32.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:33:32.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:33:32.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:33:32.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:33:32.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:33:32.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:33:32.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:33:32.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-09-10T20:33:32.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:33:32.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:33:32.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:33:32.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:33:32.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:33:32.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:33:32.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:33:32.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:33:32.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:33:32.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:33:32.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:33:32.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:33:32.843 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:33:32.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:33:32.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:33:32.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:33:32.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:33:32.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:33:32.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:33:32.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:33:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:33:33.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:33:33.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:33.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:33:33.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:33:33.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:33:33.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-09-10T20:33:33.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:33:33.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:33.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:33:33.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:33:33.856 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:33:33.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 26a72458-a663-42f7-ba97-8200ae2aad5f' 2024-09-10T20:33:33.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:33:33.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdreBmYe2uMxAATJeJnhynniasPBhVJ7N23w== 2024-09-10T20:33:33.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdreBmYe2uMxAATJeJnhynniasPBhVJ7N23w=="}' 2024-09-10T20:33:33.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26a72458-a663-42f7-ba97-8200ae2aad5f -i td/osd-scrub-repair/0/new.json 2024-09-10T20:33:34.074 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:33:34.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:33:34.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAdreBmYe2uMxAATJeJnhynniasPBhVJ7N23w== --osd-uuid 26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:33:34.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:34.112+0000 7fdc5df56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:34.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:34.120+0000 7fdc5df56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:34.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:34.121+0000 7fdc5df56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:34.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:34.122+0000 7fdc5df56640 -1 bdev(0x55d399001c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:33:34.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:34.122+0000 7fdc5df56640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:33:39.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:33:39.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:33:39.607 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:33:39.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:33:39.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:33:39.973 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:33:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:33:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:33:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:33:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:33:39.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:33:40.003 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:39.999+0000 7f1f8e9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:40.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:40.028+0000 7f1f8e9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:40.029+0000 7f1f8e9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:33:40.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:40.252 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:33:40.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:33:40.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:40.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:40.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:41.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:41.390+0000 7f1f8e9ae640 -1 Falling back to public interface 2024-09-10T20:33:41.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:41.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:41.533 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:33:41.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:33:41.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:41.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:41.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:42.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:42.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:42.819 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:33:42.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:33:42.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:42.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:43.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:43.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:33:43.310+0000 7f1f8e9ae640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:33:44.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:44.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:44.102 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:33:44.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:44.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:44.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:44.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:45.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:45.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:45.491 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:33:45.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:33:45.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:45.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:45.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:46.779 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:33:46.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:33:46.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:46.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:47.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:48.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:48.148 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:33:48.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:48.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:33:48.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:48.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:48.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:49.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:49.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:49.435 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:33:49.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:33:49.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:49.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:50.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:50.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:50.721 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:33:50.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:33:50.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:50.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:51.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:52.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:52.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:52.122 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:33:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:33:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:52.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:53.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:53.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:53.409 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:33:53.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:33:53.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:53.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:53.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:54.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:54.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:54.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:33:54.697 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:33:54.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:54.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:55.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:56.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:56.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:56.021 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:33:56.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:33:56.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:56.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:56.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:57.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:57.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:57.538 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:33:57.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:33:57.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:57.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:57.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:58.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:58.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:58.940 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:33:58.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:33:58.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:58.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:33:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:00.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:00.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:00.477 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:34:00.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:34:00.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:00.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:34:01.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:02.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:02.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:02.080 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:34:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:34:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:34:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:03.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:03.372 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:34:03.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:34:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:03.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:34:03.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:04.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:04.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:04.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:34:04.908 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:34:04.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:04.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:34:05.259 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/325140591,v1:127.0.0.1:6803/325140591] [v2:127.0.0.1:6804/325140591,v1:127.0.0.1:6805/325140591] exists,up 26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:34:05.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:34:05.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:34:05.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:34:05.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-10T20:34:05.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:34:05.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:34:05.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:34:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:34:05.267 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:34:05.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6c979eef-2d9c-4981-87a9-11649e9bbf8a' 2024-09-10T20:34:05.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:34:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9reBmmZGjEBAAN7ztxDK+jx2a9mmX1UJbDw== 2024-09-10T20:34:05.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9reBmmZGjEBAAN7ztxDK+jx2a9mmX1UJbDw=="}' 2024-09-10T20:34:05.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c979eef-2d9c-4981-87a9-11649e9bbf8a -i td/osd-scrub-repair/1/new.json 2024-09-10T20:34:05.607 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:34:05.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:34:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA9reBmmZGjEBAAN7ztxDK+jx2a9mmX1UJbDw== --osd-uuid 6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:34:05.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:05.647+0000 7f535f216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:05.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:05.688+0000 7f535f216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:05.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:05.690+0000 7f535f216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:05.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:05.690+0000 7f535f216640 -1 bdev(0x55c9fb73fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:34:05.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:05.690+0000 7f535f216640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:34:12.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:34:12.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:34:12.124 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:34:12.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:34:12.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:34:12.486 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:34:12.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:34:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:34:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:34:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:34:12.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:34:12.516 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:12.512+0000 7fa8ed3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:12.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:12.514+0000 7fa8ed3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:12.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:12.515+0000 7fa8ed3b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:12.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:34:12.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:34:12.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:34:12.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:34:12.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:34:12.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:12.763 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:34:12.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:34:12.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:12.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:13.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:14.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:14.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:14.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:34:14.042 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:34:14.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:14.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:14.131+0000 7fa8ed3b2640 -1 Falling back to public interface 2024-09-10T20:34:14.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:15.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:15.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:15.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:34:15.326 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:34:15.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:15.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:15.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:16.334 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:16.330+0000 7fa8ed3b2640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:34:16.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:16.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:16.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:34:16.618 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:34:16.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:16.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:17.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:18.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:18.012 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:34:18.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:18.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:34:18.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:18.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:18.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:19.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:19.293 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:34:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:34:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:19.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:20.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:20.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:20.582 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:34:20.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:34:20.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:20.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:20.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:21.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:21.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:21.873 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:34:21.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:34:21.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:21.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:22.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:23.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:23.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:23.288 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:34:23.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:34:23.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:23.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:23.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:24.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:24.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:24.578 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:34:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:34:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:24.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:24.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:25.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:25.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:25.870 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:34:25.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:34:25.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:25.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:26.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:27.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:27.244 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:34:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:34:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:27.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:28.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:28.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:28.534 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:34:28.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:34:28.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:28.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:29.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:30.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:30.313 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:34:30.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:30.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:34:30.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:30.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:30.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:31.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:31.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:31.884 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:34:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:34:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:32.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:33.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:33.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:33.334 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:34:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:34:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:33.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:33.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:34.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:34.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:34.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:34:34.747 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:34:34.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:34.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:35.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:36.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:36.382 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:34:36.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:36.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:34:36.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:36.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:37.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:38.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:38.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:38.059 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:34:38.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:34:38.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:38.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:38.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:39.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:39.437 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:34:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:34:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:39.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:34:39.724 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4006612186,v1:127.0.0.1:6811/4006612186] [v2:127.0.0.1:6812/4006612186,v1:127.0.0.1:6813/4006612186] exists,up 6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:34:39.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:34:39.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:34:39.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:34:39.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:34:39.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8e853b5e-1b74-4089-b7ed-f5a8915cdee1 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8e853b5e-1b74-4089-b7ed-f5a8915cdee1' 2024-09-10T20:34:39.732 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 8e853b5e-1b74-4089-b7ed-f5a8915cdee1 2024-09-10T20:34:39.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:34:39.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBfreBmNWpALBAAuu7j/GlOzipCQjb25AzPFw== 2024-09-10T20:34:39.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBfreBmNWpALBAAuu7j/GlOzipCQjb25AzPFw=="}' 2024-09-10T20:34:39.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8e853b5e-1b74-4089-b7ed-f5a8915cdee1 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:34:40.110 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:34:40.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:34:40.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBfreBmNWpALBAAuu7j/GlOzipCQjb25AzPFw== --osd-uuid 8e853b5e-1b74-4089-b7ed-f5a8915cdee1 2024-09-10T20:34:40.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:40.150+0000 7f921400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:40.194 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:40.191+0000 7f921400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:40.195 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:40.192+0000 7f921400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:40.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:40.192+0000 7f921400b640 -1 bdev(0x55e172b03c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:34:40.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:40.192+0000 7f921400b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:34:46.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:34:46.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:34:46.133 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:34:46.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:34:46.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:34:46.492 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:34:46.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:34:46.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:34:46.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:34:46.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:34:46.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:34:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:46.518+0000 7f46799c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:46.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:46.520+0000 7f46799c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:46.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:46.521+0000 7f46799c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:34:46.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:34:46.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:34:46.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:34:46.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:34:46.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:34:46.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:46.768 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:34:46.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:34:46.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:46.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:47.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:47.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:47.663+0000 7f46799c5640 -1 Falling back to public interface 2024-09-10T20:34:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:48.046 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:34:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:34:48.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:48.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:48.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:49.225 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:34:49.221+0000 7f46799c5640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:34:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:49.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:49.327 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:34:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:34:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:49.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:49.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:50.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:50.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:50.710 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:34:50.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:34:50.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:50.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:51.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:52.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:52.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:52.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:34:52.005 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:34:52.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:52.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:52.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:53.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:53.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:53.293 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:34:53.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:34:53.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:53.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:53.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:54.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:54.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:54.580 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:34:54.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:34:54.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:54.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:54.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:55.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:55.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:55.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:34:55.871 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:34:55.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:55.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:56.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:57.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:57.295 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:34:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:34:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:57.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:57.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:58.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:58.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:58.583 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:34:58.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:34:58.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:58.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:34:58.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:34:59.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:34:59.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:34:59.875 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:34:59.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:34:59.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:34:59.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:00.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:01.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:01.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:01.439 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:35:01.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:35:01.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:01.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:01.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:02.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:02.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:02.729 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:35:02.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:35:02.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:02.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:03.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:04.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:04.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:04.204 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:35:04.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:35:04.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:04.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:05.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:05.903 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:35:05.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:05.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:35:05.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:05.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:06.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:07.650 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:35:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:35:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:07.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:08.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:09.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:09.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:09.404 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:35:09.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:35:09.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:09.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:09.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:10.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:10.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:10.898 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:35:10.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:35:10.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:10.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:12.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:12.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:12.330 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:35:12.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:35:12.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:12.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:12.660 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2402273626,v1:127.0.0.1:6819/2402273626] [v2:127.0.0.1:6820/2402273626,v1:127.0.0.1:6821/2402273626] exists,up 8e853b5e-1b74-4089-b7ed-f5a8915cdee1 2024-09-10T20:35:12.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:12.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:12.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:12.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-09-10T20:35:12.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-10T20:35:13.040 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T20:35:13.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:35:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-09-10T20:35:14.640 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T20:35:14.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-10T20:35:14.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:35:14.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:35:14.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:35:14.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:35:14.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:35:14.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:35:14.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:35:14.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:35:14.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:35:14.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:35:14.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:35:14.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:35:14.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:35:14.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:35:14.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:15.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:35:15.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-10T20:35:15.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-10T20:35:15.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-10T20:35:15.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:15.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:35:15.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-10T20:35:15.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-10T20:35:15.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2024-09-10T20:35:15.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:15.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:35:15.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-10T20:35:15.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-10T20:35:15.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509447' 2024-09-10T20:35:15.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:15.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-10T20:35:15.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:15.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:35:15.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-10T20:35:15.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:15.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-10T20:35:15.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-10T20:35:15.427 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836495 2024-09-10T20:35:15.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:15.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-10T20:35:15.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:35:16.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:35:16.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:17.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-10T20:35:17.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:17.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-10T20:35:17.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:17.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:35:17.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-10T20:35:17.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:17.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-10T20:35:17.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-10T20:35:17.009 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672974 2024-09-10T20:35:17.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:35:17.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-10T20:35:17.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:17.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-10T20:35:17.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:17.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:35:17.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-10T20:35:17.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:17.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-10T20:35:17.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-10T20:35:17.287 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509447 2024-09-10T20:35:17.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:35:17.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-10T20:35:17.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:35:17.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:17.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:17.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:18.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:35:18.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:18.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:18.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-09-10T20:35:18.527 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-09-10T20:35:18.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:18.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:18.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:18.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:18.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:18.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:18.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-10T20:35:19.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-10T20:35:19.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-10T20:35:19.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:35:19.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:35:19.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:35:19.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:35:19.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:35:19.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:19.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:35:20.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:35:20.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:20.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:35:20.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:35:20.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:35:20.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:35:20.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:35:20.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:35:20.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:35:20.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:35:20.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-10T20:35:22.373 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T20:35:22.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:22.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:35:22.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:35:22.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:35:22.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:22.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:22.928 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:22.924+0000 7fd973c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:22.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:22.927+0000 7fd973c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:22.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:22.928+0000 7fd973c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:35:23.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:23.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:24.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:24.025+0000 7fd973c0b640 -1 Falling back to public interface 2024-09-10T20:35:24.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:24.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:24.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:25.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:25.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:25.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:25.745 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:35:25.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:25.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:25.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:25.850+0000 7fd973c0b640 -1 osd.0 20 log_to_monitors true 2024-09-10T20:35:26.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:27.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/3219083986,v1:127.0.0.1:6803/3219083986] [v2:127.0.0.1:6804/3219083986,v1:127.0.0.1:6805/3219083986] exists,up 26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:35:27.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:35:27.376 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:35:27.376 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:35:27.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:35:27.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:35:27.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:35:27.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:35:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:35:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:35:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:35:27.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:35:27.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:35:27.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:35:27.737 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:27.738 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:35:27.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:35:27.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:27.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:35:27.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-10T20:35:27.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-10T20:35:27.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-10T20:35:27.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:27.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:35:27.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-10T20:35:27.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-10T20:35:27.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672977' 2024-09-10T20:35:27.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:27.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672977 2-64424509451' 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-10T20:35:28.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:28.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:35:28.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-10T20:35:28.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:28.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-10T20:35:28.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-10T20:35:28.051 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 103079215106 2024-09-10T20:35:28.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:28.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-10T20:35:28.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:35:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:35:29.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-10T20:35:29.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:35:30.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:35:30.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-10T20:35:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:30.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-10T20:35:30.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:30.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:35:30.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-10T20:35:30.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:30.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-10T20:35:30.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-10T20:35:30.905 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672977 2024-09-10T20:35:30.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:35:31.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-10T20:35:31.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:31.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T20:35:31.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:31.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:35:31.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T20:35:31.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:31.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T20:35:31.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T20:35:31.182 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509451 2024-09-10T20:35:31.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:35:31.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-10T20:35:31.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:35:31.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:31.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:31.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:35:31.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:31.802 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:31.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:31.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:31.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:31.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:31.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:32.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:35:32.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:32.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:32.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:32.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:35:32.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:32.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:35:32.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:35:32.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:35:32.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:35:32.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:35:32.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:35:32.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:35:33.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-09-10T20:35:35.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:35.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:35:35.085 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:35.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:35.111+0000 7fde4f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:35.118 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:35.114+0000 7fde4f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:35.119 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:35.116+0000 7fde4f9bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:35.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:36.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:36.237+0000 7fde4f9bb640 -1 Falling back to public interface 2024-09-10T20:35:36.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:36.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:36.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:36.647 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:36.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:36.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:36.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:37.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:37.574+0000 7fde4f9bb640 -1 osd.1 25 log_to_monitors true 2024-09-10T20:35:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:35:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:37.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:38.321 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/3036476421,v1:127.0.0.1:6811/3036476421] [v2:127.0.0.1:6812/3036476421,v1:127.0.0.1:6813/3036476421] exists,up 6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:35:38.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:35:38.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:35:38.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:35:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:35:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:38.686 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:35:38.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:35:38.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:38.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:35:38.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215109 2024-09-10T20:35:38.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215109 2024-09-10T20:35:38.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109' 2024-09-10T20:35:38.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:38.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:35:38.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-10T20:35:38.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-10T20:35:38.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2024-09-10T20:35:38.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:38.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:35:39.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-10T20:35:39.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-10T20:35:39.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109 1-124554051586 2-64424509454' 2024-09-10T20:35:39.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:39.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215109 2024-09-10T20:35:39.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:39.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:35:39.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215109 2024-09-10T20:35:39.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:39.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215109 2024-09-10T20:35:39.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-09-10T20:35:39.030 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 103079215109 2024-09-10T20:35:39.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:39.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215108 -lt 103079215109 2024-09-10T20:35:39.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:35:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:35:40.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:40.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215109 -lt 103079215109 2024-09-10T20:35:40.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:40.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-10T20:35:40.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:40.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:35:40.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-10T20:35:40.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:40.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-10T20:35:40.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-10T20:35:40.595 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 124554051586 2024-09-10T20:35:40.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:35:40.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-10T20:35:40.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:40.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-10T20:35:40.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:40.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:35:40.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-10T20:35:40.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:40.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-10T20:35:40.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-10T20:35:40.875 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509454 2024-09-10T20:35:40.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:35:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-10T20:35:41.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:35:41.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:41.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:41.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:35:41.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:41.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:41.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:41.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:41.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:41.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:41.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:41.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:35:41.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:41.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:41.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:35:42.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:42.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:42.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:42.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:42.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:35:42.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:42.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:42.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:42.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:35:42.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:35:42.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:35:42.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:35:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:35:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:35:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:35:42.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:35:42.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:35:43.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:35:43.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:35:43.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:35:43.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:35:43.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:35:43.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:35:43.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:43.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:43.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:43.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:43.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:43.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:43.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:43.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:35:43.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:43.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:43.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:35:43.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:35:43.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:35:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:35:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:35:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T20:35:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:35:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:35:44.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:44.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:45.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:35:45.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:45.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:45.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:45.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:35:45.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:35:45.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:45.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:35:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:35:45.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:35:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:35:46.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-10T20:35:47.527 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:104778fc:::obj2:head# 2024-09-10T20:35:48.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:48.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:35:48.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:48.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:35:48.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:48.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:48.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:35:48.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:35:48.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:35:48.048 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:35:48.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:48.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:35:48.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:35:48.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:48.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:48.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:48.078 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:48.074+0000 7f58a1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:48.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:48.077+0000 7f58a1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:48.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:48.078+0000 7f58a1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:48.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:35:48.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:48.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:35:48.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:48.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:48.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:49.427 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:49.423+0000 7f58a1211640 -1 Falling back to public interface 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:49.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:49.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:50.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:35:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:51.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:51.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:51.513+0000 7f58a1211640 -1 osd.0 31 log_to_monitors true 2024-09-10T20:35:52.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:52.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:52.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:35:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:35:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:52.493 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,31) [v2:127.0.0.1:6802/2400943559,v1:127.0.0.1:6803/2400943559] [v2:127.0.0.1:6804/2400943559,v1:127.0.0.1:6805/2400943559] exists,up 26a72458-a663-42f7-ba97-8200ae2aad5f 2024-09-10T20:35:52.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:52.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:52.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:52.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:35:52.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:35:52.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:35:52.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:35:52.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:35:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:35:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:35:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:35:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:35:52.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:35:52.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:35:52.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:35:52.859 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:35:52.859 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:35:52.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:35:52.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:52.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:35:52.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-10T20:35:52.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-10T20:35:52.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066' 2024-09-10T20:35:52.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:52.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:35:53.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051590 2024-09-10T20:35:53.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051590 2024-09-10T20:35:53.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-124554051590' 2024-09-10T20:35:53.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:53.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:35:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T20:35:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T20:35:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-124554051590 2-64424509458' 2024-09-10T20:35:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888066 2024-09-10T20:35:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:53.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:35:53.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888066 2024-09-10T20:35:53.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-10T20:35:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-09-10T20:35:53.170 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 146028888066 2024-09-10T20:35:53.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:53.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-10T20:35:53.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:35:54.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:35:54.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:35:54.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-10T20:35:54.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:54.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051590 2024-09-10T20:35:54.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:54.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:35:54.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051590 2024-09-10T20:35:54.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:54.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051590 2024-09-10T20:35:54.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051590' 2024-09-10T20:35:54.731 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 124554051590 2024-09-10T20:35:54.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:35:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051590 2024-09-10T20:35:55.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:35:55.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T20:35:55.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:35:55.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:35:55.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T20:35:55.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:35:55.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T20:35:55.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T20:35:55.008 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509458 2024-09-10T20:35:55.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:35:55.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-10T20:35:55.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:35:55.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:55.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:55.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:35:55.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:35:55.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:35:55.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:35:55.626 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:35:55.626 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:35:55.626 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:35:55.626 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:35:55.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:35:55.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:35:55.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:35:55.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:35:56.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:35:56.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:35:56.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:35:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:35:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:35:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:35:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:35:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:35:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:35:56.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:35:56.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-10T20:35:58.570 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-09-10T20:35:59.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:59.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:59.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:35:59.090 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:59.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:59.121 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:59.117+0000 7f12bc60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:59.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:59.119+0000 7f12bc60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:59.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:35:59.121+0000 7f12bc60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:59.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:00.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:00.245+0000 7f12bc60e640 -1 Falling back to public interface 2024-09-10T20:36:00.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:00.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:00.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:36:00.655 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:36:00.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:00.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:36:00.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:01.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:01.773+0000 7f12bc60e640 -1 osd.1 35 log_to_monitors true 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:01.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:36:02.288 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 39 up_thru 34 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/1412160195,v1:127.0.0.1:6811/1412160195] [v2:127.0.0.1:6812/1412160195,v1:127.0.0.1:6813/1412160195] exists,up 6c979eef-2d9c-4981-87a9-11649e9bbf8a 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:36:02.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:36:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:36:02.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:36:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:36:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:36:02.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:36:02.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:36:02.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:36:02.691 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:36:02.691 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:36:02.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:36:02.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:02.691 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:36:02.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888069 2024-09-10T20:36:02.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888069 2024-09-10T20:36:02.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888069' 2024-09-10T20:36:02.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:02.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:36:02.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724546 2024-09-10T20:36:02.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724546 2024-09-10T20:36:02.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2024-09-10T20:36:02.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:02.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-64424509461' 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888069 2024-09-10T20:36:03.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:36:03.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888069 2024-09-10T20:36:03.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:03.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888069 2024-09-10T20:36:03.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-09-10T20:36:03.082 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 146028888069 2024-09-10T20:36:03.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888068 -lt 146028888069 2024-09-10T20:36:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:36:04.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:36:04.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:04.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-10T20:36:04.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:04.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724546 2024-09-10T20:36:04.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:04.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:36:04.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724546 2024-09-10T20:36:04.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724546 2024-09-10T20:36:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-09-10T20:36:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 167503724546 2024-09-10T20:36:04.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:36:04.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724546 -lt 167503724546 2024-09-10T20:36:04.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:04.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-10T20:36:04.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:04.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:36:04.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-10T20:36:04.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:04.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-10T20:36:04.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-10T20:36:04.923 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509461 2024-09-10T20:36:04.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:36:05.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-10T20:36:05.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:36:05.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:05.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:36:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:36:05.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:36:05.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:36:05.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:36:05.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:36:05.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:36:05.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:36:05.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:36:05.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:36:05.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:05.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:06.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:36:06.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:36:06.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:36:06.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-10T20:36:06.258 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T20:36:06.258 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T20:36:06.259 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:36:06.272 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T20:36:06.272 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T20:36:06.272 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:36:06.285 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T20:36:06.285 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-10T20:36:06.285 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:36:06.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-09-10T20:36:06.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T20:36:06.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-10T20:36:06.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-10T20:36:06.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:36:06.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-09-10T20:36:06.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-09-10T20:36:06.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:36:06.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T20:36:06.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T20:36:06.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:36:06.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-09-10T20:36:06.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-09-10T20:36:06.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:06.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:06.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:06.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:07.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:07.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-09-10T20:36:07.263 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:36:07.263 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T20:36:07.263 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T20:36:07.263 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-27T20:34:27.259823+0000" 2024-09-10T20:36:07.263 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:36:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:07.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T20:36:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:36:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:36:07.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:07.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:07.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:07.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:07.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:07.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:07.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:35:12.977358+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:07.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:36:08.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:08.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:08.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:08.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:08.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:08.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:08.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:35:12.977358+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:08.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:36:09.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:36:09.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:09.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:09.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:09.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:09.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:09.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:10.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:35:12.977358+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:10.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:36:11.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:36:11.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:11.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:11.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:11.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:11.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:11.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:11.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:35:12.977358+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:11.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:36:12.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:36:12.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:12.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:12.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:12.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:12.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:12.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:12.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:35:12.977358+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:12.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:36:13.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:36:13.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:36:13.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:36:13.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:36:13.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:36:13.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:36:13.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:36:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:36:08.754977+0000 '>' 2024-09-10T20:35:12.977358+0000 2024-09-10T20:36:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:36:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-10T20:36:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:36:13.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:36:13.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:36:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:36:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:36:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:36:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:36:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:36:14.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:36:14.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:36:14.328 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:36:14.328 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:36:14.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:36:14.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:14.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:36:14.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888072 2024-09-10T20:36:14.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888072 2024-09-10T20:36:14.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888072' 2024-09-10T20:36:14.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:14.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:36:14.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724549 2024-09-10T20:36:14.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724549 2024-09-10T20:36:14.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888072 1-167503724549' 2024-09-10T20:36:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:14.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:36:14.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-10T20:36:14.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-10T20:36:14.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888072 1-167503724549 2-64424509464' 2024-09-10T20:36:14.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:14.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888072 2024-09-10T20:36:14.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:14.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:36:14.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888072 2024-09-10T20:36:14.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:14.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888072 2024-09-10T20:36:14.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888072' 2024-09-10T20:36:14.639 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 146028888072 2024-09-10T20:36:14.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888071 -lt 146028888072 2024-09-10T20:36:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:36:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:36:15.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:16.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888072 -lt 146028888072 2024-09-10T20:36:16.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:16.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724549 2024-09-10T20:36:16.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:36:16.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724549 2024-09-10T20:36:16.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:16.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724549 2024-09-10T20:36:16.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-09-10T20:36:16.214 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 167503724549 2024-09-10T20:36:16.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:36:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724549 -lt 167503724549 2024-09-10T20:36:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:16.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-10T20:36:16.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:16.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:36:16.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-10T20:36:16.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:16.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-10T20:36:16.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-10T20:36:16.495 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509464 2024-09-10T20:36:16.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:36:16.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-10T20:36:16.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:36:16.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:16.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:36:17.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:36:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:36:17.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:36:17.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:17.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:36:17.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:18.037 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:36:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888074 2024-09-10T20:36:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888074 2024-09-10T20:36:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888074' 2024-09-10T20:36:18.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:18.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:36:18.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724551 2024-09-10T20:36:18.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724551 2024-09-10T20:36:18.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888074 1-167503724551' 2024-09-10T20:36:18.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:18.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:36:18.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-10T20:36:18.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-10T20:36:18.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888074 1-167503724551 2-64424509466' 2024-09-10T20:36:18.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:18.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888074 2024-09-10T20:36:18.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:18.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:36:18.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888074 2024-09-10T20:36:18.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:18.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888074 2024-09-10T20:36:18.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888074' 2024-09-10T20:36:18.349 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 146028888074 2024-09-10T20:36:18.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:18.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888073 -lt 146028888074 2024-09-10T20:36:18.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:36:19.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:36:19.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:19.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888073 -lt 146028888074 2024-09-10T20:36:19.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:36:20.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:36:20.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:21.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888074 -lt 146028888074 2024-09-10T20:36:21.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:21.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724551 2024-09-10T20:36:21.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:21.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:36:21.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724551 2024-09-10T20:36:21.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:21.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724551 2024-09-10T20:36:21.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2024-09-10T20:36:21.201 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 167503724551 2024-09-10T20:36:21.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:36:21.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724551 -lt 167503724551 2024-09-10T20:36:21.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:21.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-10T20:36:21.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:21.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:36:21.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-10T20:36:21.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:21.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-10T20:36:21.479 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509466 2024-09-10T20:36:21.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-10T20:36:21.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:36:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-10T20:36:21.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-09-10T20:36:21.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-10T20:36:22.018 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:36:22.018 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-10T20:36:08.755089+0000 20'10 40:107 [1,0] 1 [1,0] 1 20'10 2024-09-10T20:36:08.754977+0000 20'10 2024-09-10T20:36:08.754977+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:36:08.754977+0000 10 0 2024-09-10T20:36:22.018 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:36:22.018 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:36:22.018 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:36:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-10T20:36:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-09-10T20:36:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:36:22.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:36:22.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:36:22.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:36:22.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:36:22.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:36:22.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:36:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:36:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:36:22.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:36:22.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:22.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:36:22.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:36:22.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:22.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:36:22.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:36:22.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:36:22.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:36:22.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:22.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:22.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:36:22.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:36:22.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:36:22.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:36:22.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:36:22.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:36:22.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:22.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:36:22.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:36:22.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:36:22.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:36:22.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:22.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:22.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:36:22.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:36:22.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:36:22.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:36:22.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:36:22.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:22.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:22.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-09-10T20:36:22.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:36:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:36:22.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:22.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:36:23.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:36:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:36:23.055 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:36:23.055 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:23.055 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:23.056 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:36:23.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:36:23.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:36:23.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:36:23.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:36:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:36:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:36:24.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:36:24.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-09-10T20:36:24.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-10T20:36:24.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:36:24.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:36:24.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:36:24.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:36:24.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:36:24.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8b44be36-0ec2-490b-be9f-058b413d9a8c' 2024-09-10T20:36:24.068 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:36:24.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:36:24.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIreBmC0/FBBAA5eHcGnzN5R3Jkm707KpUaQ== 2024-09-10T20:36:24.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIreBmC0/FBBAA5eHcGnzN5R3Jkm707KpUaQ=="}' 2024-09-10T20:36:24.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8b44be36-0ec2-490b-be9f-058b413d9a8c -i td/osd-scrub-repair/0/new.json 2024-09-10T20:36:24.328 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:36:24.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:36:24.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDIreBmC0/FBBAA5eHcGnzN5R3Jkm707KpUaQ== --osd-uuid 8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:36:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:24.368+0000 7fbba100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:24.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:24.370+0000 7fbba100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:24.375 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:24.371+0000 7fbba100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:24.372+0000 7fbba100b640 -1 bdev(0x55bd8fca7c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:36:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:24.372+0000 7fbba100b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:36:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:36:30.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:36:30.626 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:36:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:36:30.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:36:30.983 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:36:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:36:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:36:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:36:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:36:30.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:36:31.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:31.010+0000 7fe45c2eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:31.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:31.013+0000 7fe45c2eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:31.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:31.014+0000 7fe45c2eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:31.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:36:31.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:36:31.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:31.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:31.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:32.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:32.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:32.532 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:36:32.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:36:32.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:32.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:32.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:32.623+0000 7fe45c2eb640 -1 Falling back to public interface 2024-09-10T20:36:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:33.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:33.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:33.816 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:36:33.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:36:33.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:33.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:34.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:34.500 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:34.496+0000 7fe45c2eb640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:36:35.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:35.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:35.098 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:36:35.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:36:35.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:35.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:35.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:36.437 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:36.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:37.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:37.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:37.725 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:36:37.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:36:37.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:37.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:38.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:39.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:39.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:39.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:36:39.056 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:36:39.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:39.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:40.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:36:40.434 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:40.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:40.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:41.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:41.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:41.721 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:36:41.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:36:41.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:41.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:42.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:43.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:43.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:43.047 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:36:43.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:36:43.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:43.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:43.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:44.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:44.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:44.428 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:36:44.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:36:44.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:44.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:44.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:45.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:45.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:45.714 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:36:45.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:36:45.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:45.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:46.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:47.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:47.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:47.229 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:36:47.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:36:47.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:47.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:47.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:48.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:48.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:48.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:36:48.689 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:36:48.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:48.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:49.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:50.391 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:36:50.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:36:50.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:50.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:51.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:52.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:52.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:52.161 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:36:52.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:36:52.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:52.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:52.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:53.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:53.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:53.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:36:53.521 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:36:53.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:53.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:54.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:55.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:55.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:55.190 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:36:55.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:36:55.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:55.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:55.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:56.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:56.951 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:36:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:36:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:56.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:57.236 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3798210295,v1:127.0.0.1:6803/3798210295] [v2:127.0.0.1:6804/3798210295,v1:127.0.0.1:6805/3798210295] exists,up 8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:36:57.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:36:57.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:36:57.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a6d260f9-d264-453a-b583-f24c23635a1f 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a6d260f9-d264-453a-b583-f24c23635a1f' 2024-09-10T20:36:57.243 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 a6d260f9-d264-453a-b583-f24c23635a1f 2024-09-10T20:36:57.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:36:57.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpreBmG0wqDxAAwNJCfxLwUZDnYWnsgVBhIQ== 2024-09-10T20:36:57.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpreBmG0wqDxAAwNJCfxLwUZDnYWnsgVBhIQ=="}' 2024-09-10T20:36:57.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a6d260f9-d264-453a-b583-f24c23635a1f -i td/osd-scrub-repair/1/new.json 2024-09-10T20:36:57.609 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:36:57.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:36:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDpreBmG0wqDxAAwNJCfxLwUZDnYWnsgVBhIQ== --osd-uuid a6d260f9-d264-453a-b583-f24c23635a1f 2024-09-10T20:36:57.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:57.650+0000 7ffa74242640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:57.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:57.689+0000 7ffa74242640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:57.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:57.690+0000 7ffa74242640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:57.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:57.690+0000 7ffa74242640 -1 bdev(0x560f62ca9c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:36:57.695 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:36:57.690+0000 7ffa74242640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:37:04.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:37:04.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:37:04.086 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:37:04.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:37:04.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:37:04.444 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:37:04.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:37:04.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:37:04.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:37:04.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:37:04.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:37:04.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:04.470+0000 7f3170020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:04.476 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:04.473+0000 7f3170020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:04.477 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:04.474+0000 7f3170020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:37:04.725 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:37:04.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:04.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:06.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:06.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:06.002 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:37:06.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:37:06.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:06.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:06.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:06.077+0000 7f3170020640 -1 Falling back to public interface 2024-09-10T20:37:06.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:07.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:07.293 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:37:07.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:37:07.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:07.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:07.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:07.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:07.695+0000 7f3170020640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:37:08.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:08.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:08.576 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:37:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:37:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:08.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:09.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:09.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:09.859 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:37:09.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:37:09.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:09.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:10.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:11.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:11.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:11.146 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:37:11.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:37:11.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:11.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:12.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:12.538 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:37:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:37:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:12.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:12.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:13.865 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:37:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:37:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:14.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:15.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:15.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:15.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:37:15.156 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:37:15.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:15.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:15.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:16.574 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:37:16.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:37:16.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:16.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:16.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:17.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:17.956 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:37:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:37:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:17.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:18.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:19.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:19.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:19.272 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:37:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:37:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:19.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:20.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:21.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:21.062 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:37:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:37:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:21.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:21.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:22.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:22.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:22.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:37:22.712 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:37:22.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:22.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:23.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:24.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:24.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:24.658 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:37:24.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:37:24.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:24.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:26.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:26.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:26.350 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:37:26.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:37:26.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:26.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:26.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:27.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:27.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:27.979 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:37:27.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:37:27.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:27.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:28.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:29.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:29.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:29.697 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:37:29.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:37:29.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:29.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:30.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:31.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:31.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:31.091 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:37:31.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:37:31.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:31.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:37:31.381 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/802518762,v1:127.0.0.1:6811/802518762] [v2:127.0.0.1:6812/802518762,v1:127.0.0.1:6813/802518762] exists,up a6d260f9-d264-453a-b583-f24c23635a1f 2024-09-10T20:37:31.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:37:31.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:31.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:37:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3e699987-84fb-449d-a6fc-5d44a699edc6 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 3e699987-84fb-449d-a6fc-5d44a699edc6 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3e699987-84fb-449d-a6fc-5d44a699edc6' 2024-09-10T20:37:31.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:37:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALruBmowGxFxAAMOSq79vYmUo43EMIlAyJbw== 2024-09-10T20:37:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALruBmowGxFxAAMOSq79vYmUo43EMIlAyJbw=="}' 2024-09-10T20:37:31.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3e699987-84fb-449d-a6fc-5d44a699edc6 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:37:31.756 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:37:31.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:37:31.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQALruBmowGxFxAAMOSq79vYmUo43EMIlAyJbw== --osd-uuid 3e699987-84fb-449d-a6fc-5d44a699edc6 2024-09-10T20:37:31.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:31.799+0000 7f01fc31e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:31.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:31.820+0000 7f01fc31e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:31.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:31.821+0000 7f01fc31e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:31.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:31.821+0000 7f01fc31e640 -1 bdev(0x55b68faa7c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:31.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:31.822+0000 7f01fc31e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:37:37.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:37:37.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:37:37.496 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:37:37.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:37:37.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:37:37.855 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:37:37.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:37:37.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:37:37.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:37:37.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:37:37.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:37:37.886 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:37.882+0000 7f8ab6a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:37.888 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:37.884+0000 7f8ab6a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:37.889 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:37.885+0000 7f8ab6a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:38.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:37:38.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:37:38.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:38.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:38.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:39.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:39.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:39.422 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:37:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:37:39.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:39.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:39.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:39.505+0000 7f8ab6a11640 -1 Falling back to public interface 2024-09-10T20:37:39.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:40.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:40.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:40.705 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:37:40.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:37:40.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:40.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:40.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:41.571 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:37:41.567+0000 7f8ab6a11640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:37:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:41.990 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:37:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:37:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:41.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:42.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:43.341 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:37:43.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:37:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:43.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:43.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:44.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:44.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:44.632 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:37:44.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:37:44.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:44.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:44.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:45.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:45.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:45.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:37:45.917 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:37:45.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:45.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:46.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:47.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:47.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:47.207 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:37:47.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:37:47.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:47.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:47.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:48.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:48.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:48.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:37:48.588 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:37:48.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:48.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:48.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:49.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:49.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:49.872 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:37:49.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:37:49.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:49.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:50.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:51.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:51.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:51.162 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:37:51.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:37:51.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:51.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:51.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:52.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:52.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:52.448 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:37:52.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:37:52.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:52.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:52.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:53.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:53.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:53.737 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:37:53.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:37:53.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:53.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:54.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:55.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:55.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:55.030 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:37:55.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:37:55.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:55.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:55.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:56.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:56.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:56.757 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:37:56.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:37:56.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:56.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:57.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:58.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:58.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:58.400 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:37:58.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:37:58.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:58.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:37:58.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:59.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:59.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:59.842 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:37:59.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:37:59.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:59.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:38:00.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:01.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:01.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:01.368 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:38:01.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:38:01.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:01.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:38:01.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:02.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:02.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:02.890 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:38:02.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:38:02.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:02.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:38:03.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:04.526 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:38:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:38:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:38:04.812 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/151830752,v1:127.0.0.1:6819/151830752] [v2:127.0.0.1:6820/151830752,v1:127.0.0.1:6821/151830752] exists,up 3e699987-84fb-449d-a6fc-5d44a699edc6 2024-09-10T20:38:04.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:38:04.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:38:04.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:38:04.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-09-10T20:38:04.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-10T20:38:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T20:38:05.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:38:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-09-10T20:38:06.600 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T20:38:06.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-10T20:38:06.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:38:06.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:38:06.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:38:06.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:38:06.613 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:38:06.614 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:38:06.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:38:06.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:38:06.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:38:06.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:38:06.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:38:06.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:38:06.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:38:06.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:38:06.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:38:07.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:38:07.081 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:07.081 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:38:07.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:38:07.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:07.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:38:07.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-10T20:38:07.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-10T20:38:07.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-10T20:38:07.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:07.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:38:07.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705678 2024-09-10T20:38:07.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705678 2024-09-10T20:38:07.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-38654705678' 2024-09-10T20:38:07.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:07.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:38:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542149 2024-09-10T20:38:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542149 2024-09-10T20:38:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-38654705678 2-60129542149' 2024-09-10T20:38:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:07.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-10T20:38:07.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:07.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:38:07.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-10T20:38:07.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-10T20:38:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-10T20:38:07.491 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836507 2024-09-10T20:38:07.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:07.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-10T20:38:07.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:38:08.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:38:08.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-10T20:38:09.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:09.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705678 2024-09-10T20:38:09.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:09.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:38:09.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705678 2024-09-10T20:38:09.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705678 2024-09-10T20:38:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2024-09-10T20:38:09.058 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 38654705678 2024-09-10T20:38:09.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:38:09.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705678 -lt 38654705678 2024-09-10T20:38:09.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:09.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542149 2024-09-10T20:38:09.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:09.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:38:09.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542149 2024-09-10T20:38:09.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:09.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542149 2024-09-10T20:38:09.377 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 60129542149 2024-09-10T20:38:09.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2024-09-10T20:38:09.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:38:09.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542149 -lt 60129542149 2024-09-10T20:38:09.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:38:09.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:09.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:09.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:38:09.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:38:09.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:38:09.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:38:09.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:38:09.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:38:09.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:38:09.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:38:10.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:38:10.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:38:10.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:10.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:10.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:38:10.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:38:10.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:38:10.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-09-10T20:38:10.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-09-10T20:38:10.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:38:10.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-10T20:38:10.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:38:10.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:38:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:38:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:38:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:38:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:38:11.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:38:11.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:11.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:38:11.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:38:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:38:11.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:38:11.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:38:11.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:38:11.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T20:38:13.231 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:38:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:38:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:38:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:38:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:38:13.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:13.779+0000 7f8bc59fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:13.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:13.782+0000 7f8bc59fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:13.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:13.783+0000 7f8bc59fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:38:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:14.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:14.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:14.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:14.904+0000 7f8bc59fd640 -1 Falling back to public interface 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:15.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:15.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:16.595 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:16.591+0000 7f8bc59fd640 -1 osd.0 20 log_to_monitors true 2024-09-10T20:38:16.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:16.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:16.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:38:16.606 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:38:16.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:16.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:16.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:17.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1632428167,v1:127.0.0.1:6803/1632428167] [v2:127.0.0.1:6804/1632428167,v1:127.0.0.1:6805/1632428167] exists,up 8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:38:18.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:38:18.189 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:38:18.189 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:38:18.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:38:18.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:38:18.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:38:18.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:38:18.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:38:18.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:38:18.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:38:18.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:38:18.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:18.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:38:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-10T20:38:18.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-10T20:38:18.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-10T20:38:18.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:18.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:38:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705681 2024-09-10T20:38:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705681 2024-09-10T20:38:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-38654705681' 2024-09-10T20:38:18.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:18.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542152 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542152 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-38654705681 2-60129542152' 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-10T20:38:18.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:18.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:38:18.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-10T20:38:18.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:18.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-10T20:38:18.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-10T20:38:18.897 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 103079215106 2024-09-10T20:38:18.897 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:19.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-10T20:38:19.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:38:20.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:38:20.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-10T20:38:20.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:20.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705681 2024-09-10T20:38:20.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:20.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:38:20.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705681 2024-09-10T20:38:20.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:20.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705681 2024-09-10T20:38:20.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2024-09-10T20:38:20.466 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 38654705681 2024-09-10T20:38:20.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:38:20.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705681 -lt 38654705681 2024-09-10T20:38:20.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:20.743 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542152 2024-09-10T20:38:20.743 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:20.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:38:20.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542152 2024-09-10T20:38:20.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:20.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542152 2024-09-10T20:38:20.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2024-09-10T20:38:20.746 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 60129542152 2024-09-10T20:38:20.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:38:21.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542152 -lt 60129542152 2024-09-10T20:38:21.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:38:21.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:21.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:21.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:38:21.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:38:21.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:38:21.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:38:21.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:21.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:21.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:38:22.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-09-10T20:38:22.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-09-10T20:38:22.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:38:22.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:38:22.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:22.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:38:22.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-09-10T20:38:22.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-09-10T20:38:22.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:22.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:22.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:22.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:22.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:22.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-09-10T20:38:22.924 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T20:38:22.925 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T20:38:22.925 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T20:38:22.925 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T20:36:42.921759+0000" 2024-09-10T20:38:22.925 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T20:38:22.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:22.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T20:38:22.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:22.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:22.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:38:05.128517+0000 '>' 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:23.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:38:24.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:38:24.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:38:24.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:38:24.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:24.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:24.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:24.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:24.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:38:05.128517+0000 '>' 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:24.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:38:25.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:38:25.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:38:25.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:38:25.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:25.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:25.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:25.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:25.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:38:05.128517+0000 '>' 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:25.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:38:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:38:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:38:26.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:38:26.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:26.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:26.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:26.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:27.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:38:05.128517+0000 '>' 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:27.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:38:28.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:38:28.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:38:28.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T20:38:28.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:38:28.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:38:28.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:38:28.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:38:28.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:38:24.897475+0000 '>' 2024-09-10T20:38:05.128517+0000 2024-09-10T20:38:28.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:38:28.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-10T20:38:28.598 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:38:28.598 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-10T20:38:25.066475+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2024-09-10T20:38:24.897475+0000 20'1 2024-09-10T20:38:24.897475+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:38:24.897475+0000 1 0 2024-09-10T20:38:28.598 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:38:28.598 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:38:28.599 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:38:28.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-10T20:38:30.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-10T20:38:30.880 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:38:30.880 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-10T20:38:25.066475+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2024-09-10T20:38:24.897475+0000 20'1 2024-09-10T20:38:24.897475+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:38:24.897475+0000 1 0 2024-09-10T20:38:30.880 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:38:30.880 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:38:30.880 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:38:30.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-10T20:38:32.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-10T20:38:33.164 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:38:33.164 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-10T20:38:25.066475+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2024-09-10T20:38:24.897475+0000 20'1 2024-09-10T20:38:24.897475+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:38:24.897475+0000 1 0 2024-09-10T20:38:33.164 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:38:33.164 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:38:33.165 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:38:33.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-09-10T20:38:38.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-10T20:38:38.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:38:38.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:38:38.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:38:38.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:38:38.178 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:38:38.178 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:38:38.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:38:38.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:38:38.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:38:38.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:38.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:38:38.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215111 2024-09-10T20:38:38.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215111 2024-09-10T20:38:38.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215111' 2024-09-10T20:38:38.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:38.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:38:38.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705686 2024-09-10T20:38:38.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705686 2024-09-10T20:38:38.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215111 1-38654705686' 2024-09-10T20:38:38.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:38.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:38:38.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542157 2024-09-10T20:38:38.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542157 2024-09-10T20:38:38.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215111 1-38654705686 2-60129542157' 2024-09-10T20:38:38.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:38.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215111 2024-09-10T20:38:38.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:38.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:38:38.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215111 2024-09-10T20:38:38.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:38.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215111 2024-09-10T20:38:38.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2024-09-10T20:38:38.865 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 103079215111 2024-09-10T20:38:38.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:39.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215110 -lt 103079215111 2024-09-10T20:38:39.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:38:40.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:38:40.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215111 -lt 103079215111 2024-09-10T20:38:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:40.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705686 2024-09-10T20:38:40.444 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:38:40.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705686 2024-09-10T20:38:40.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705686 2024-09-10T20:38:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705686' 2024-09-10T20:38:40.447 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 38654705686 2024-09-10T20:38:40.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:38:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705686 -lt 38654705686 2024-09-10T20:38:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:40.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542157 2024-09-10T20:38:40.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:38:40.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542157 2024-09-10T20:38:40.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:40.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542157 2024-09-10T20:38:40.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2024-09-10T20:38:40.728 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 60129542157 2024-09-10T20:38:40.728 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:38:41.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542157 -lt 60129542157 2024-09-10T20:38:41.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:38:41.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:41.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:41.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:38:41.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:38:41.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:38:41.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:38:41.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:41.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:41.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:38:41.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:38:41.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:38:41.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-10T20:38:42.279 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:38:42.279 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-10T20:38:25.066475+0000 20'1 25:69 [1,0] 1 [1,0] 1 20'1 2024-09-10T20:38:24.897475+0000 20'1 2024-09-10T20:38:24.897475+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:38:24.897475+0000 1 0 2024-09-10T20:38:42.279 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:38:42.279 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:38:42.279 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:38:42.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:38:42.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:38:42.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:38:42.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:38:42.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:38:42.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:38:42.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:38:43.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-10T20:38:43.643 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:38:43.643 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:38:43.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:38:43.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:38:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-10T20:38:43.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:38:43.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:38:43.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:38:43.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:38:43.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:38:43.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:43.937+0000 7f21bae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:43.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:43.940+0000 7f21bae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:43.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:43.941+0000 7f21bae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:38:44.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:38:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:38:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:44.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:45.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:45.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:45.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:38:45.487 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:45.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:45.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:45.555+0000 7f21bae1a640 -1 Falling back to public interface 2024-09-10T20:38:45.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:46.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:47.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:38:47.432 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:47.428+0000 7f21bae1a640 -1 osd.0 25 log_to_monitors true 2024-09-10T20:38:48.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:38:48.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:38:48.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:38:48.053 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:38:48.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:38:48.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:38:48.430 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/1994453994,v1:127.0.0.1:6803/1994453994] [v2:127.0.0.1:6804/1994453994,v1:127.0.0.1:6805/1994453994] exists,up 8b44be36-0ec2-490b-be9f-058b413d9a8c 2024-09-10T20:38:48.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:38:48.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:38:48.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:38:48.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:38:48.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:38:48.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:48.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:38:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-10T20:38:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-10T20:38:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-10T20:38:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:48.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:38:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705689 2024-09-10T20:38:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705689 2024-09-10T20:38:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-38654705689' 2024-09-10T20:38:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:38:49.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:38:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542160 2024-09-10T20:38:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542160 2024-09-10T20:38:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-38654705689 2-60129542160' 2024-09-10T20:38:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:49.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-10T20:38:49.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:49.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:38:49.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-10T20:38:49.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:49.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-10T20:38:49.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-10T20:38:49.121 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 124554051586 2024-09-10T20:38:49.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:49.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-10T20:38:49.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:38:50.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:38:50.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:38:50.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-10T20:38:50.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:50.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705689 2024-09-10T20:38:50.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705689 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705689 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705689' 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 38654705689 2024-09-10T20:38:50.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:38:50.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705689 -lt 38654705689 2024-09-10T20:38:50.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:38:50.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542160 2024-09-10T20:38:50.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:38:50.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:38:50.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542160 2024-09-10T20:38:50.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:38:50.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542160 2024-09-10T20:38:50.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-09-10T20:38:50.959 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 60129542160 2024-09-10T20:38:50.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:38:51.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542160 -lt 60129542160 2024-09-10T20:38:51.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:38:51.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:51.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:38:51.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:38:51.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:38:51.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:38:51.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:38:51.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:38:51.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:38:51.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:38:52.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:38:52.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:38:52.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:38:52.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:38:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:38:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:23.905+0000 7f3154793640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:23.905+0000 7f3154793640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:23.905+0000 7f3154793640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:24.896+0000 7f3153f92640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] NEED_AUTO] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:24.896+0000 7f3153f92640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:24.896+0000 7f3153f92640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:24.896+0000 7f3153f92640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-10T20:38:52.331 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:38:24.896+0000 7f3153f92640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:38:52.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-09-10T20:38:52.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:38:52.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:38:52.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:38:52.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:38:52.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:38:52.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:38:52.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:38:52.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:38:52.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:38:52.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:38:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:38:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:38:52.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:38:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:38:52.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:38:52.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:52.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:38:52.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:38:52.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:38:52.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:38:52.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:52.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:52.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:38:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:38:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:38:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:38:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:38:52.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:38:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:38:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:38:52.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:38:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:38:52.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:38:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:38:52.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:38:52.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:38:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:52.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:38:52.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:38:52.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:38:52.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:38:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:38:52.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:38:52.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:52.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:52.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:38:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:38:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:38:52.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:38:52.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:38:52.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:52.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:52.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-09-10T20:38:52.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:38:52.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:38:53.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:38:53.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:53.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:38:53.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:38:53.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:38:53.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:38:53.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:38:53.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:38:53.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:38:53.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:38:53.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:38:53.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:38:53.193 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:38:53.194 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:53.194 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:53.194 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:38:53.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:38:53.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:38:53.273 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:38:53.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:38:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:38:54.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:38:54.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:38:54.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:38:54.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:38:54.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:54.170 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:54.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:54.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:38:54.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:38:54.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:38:54.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-09-10T20:38:54.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-10T20:38:54.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:38:54.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:38:54.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:38:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:38:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 da301d5e-e0e4-4820-91d0-37a9cf8439e2' 2024-09-10T20:38:54.207 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:38:54.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:38:54.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBeruBmiHIUDRAAKKq4lCik/zSXZoj/5LZRYg== 2024-09-10T20:38:54.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBeruBmiHIUDRAAKKq4lCik/zSXZoj/5LZRYg=="}' 2024-09-10T20:38:54.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da301d5e-e0e4-4820-91d0-37a9cf8439e2 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:38:54.474 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:38:54.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:38:54.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBeruBmiHIUDRAAKKq4lCik/zSXZoj/5LZRYg== --osd-uuid da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:38:54.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:54.513+0000 7fd60ea0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:54.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:54.521+0000 7fd60ea0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:54.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:54.522+0000 7fd60ea0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:38:54.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:54.523+0000 7fd60ea0f640 -1 bdev(0x55e72095dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:38:54.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:38:54.523+0000 7fd60ea0f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:38:59.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:38:59.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:38:59.915 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:38:59.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:38:59.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:39:00.277 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:39:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:39:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:39:00.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:39:00.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:39:00.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:00.304+0000 7fac3e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:00.310 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:00.306+0000 7fac3e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:00.311 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:00.307+0000 7fac3e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:00.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:39:00.573 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:39:00.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:00.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:00.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:39:01.847 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:39:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:02.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:02.166 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:02.163+0000 7fac3e20f640 -1 Falling back to public interface 2024-09-10T20:39:03.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:03.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:03.130 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:39:03.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:39:03.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:03.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:03.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:04.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:04.010+0000 7fac3e20f640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:39:04.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:04.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:04.427 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:39:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:39:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:04.712 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2103760242,v1:127.0.0.1:6803/2103760242] [v2:127.0.0.1:6804/2103760242,v1:127.0.0.1:6805/2103760242] exists,up da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:39:04.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:39:04.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:39:04.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:39:04.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3 2024-09-10T20:39:04.720 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3 2024-09-10T20:39:04.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3' 2024-09-10T20:39:04.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:39:04.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoruBm+p+eKxAAsopGwRqZqG+SF8Udsgwrzw== 2024-09-10T20:39:04.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoruBm+p+eKxAAsopGwRqZqG+SF8Udsgwrzw=="}' 2024-09-10T20:39:04.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3 -i td/osd-scrub-repair/1/new.json 2024-09-10T20:39:05.106 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:39:05.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:39:05.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBoruBm+p+eKxAAsopGwRqZqG+SF8Udsgwrzw== --osd-uuid 3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3 2024-09-10T20:39:05.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:05.147+0000 7fcaeb0da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:05.190 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:05.186+0000 7fcaeb0da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:05.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:05.187+0000 7fcaeb0da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:05.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:05.188+0000 7fcaeb0da640 -1 bdev(0x55bc1f6b9c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:39:05.191 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:05.188+0000 7fcaeb0da640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:39:11.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:39:11.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:39:11.465 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:39:11.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:39:11.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:39:11.829 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:39:11.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:39:11.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:11.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:39:11.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:39:11.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:39:11.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:11.856+0000 7f4e843cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:11.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:11.858+0000 7f4e843cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:11.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:11.859+0000 7f4e843cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:12.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:39:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:39:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:39:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:39:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:39:12.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:12.156 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:39:12.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:39:12.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:12.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:39:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:13.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:13.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:13.426 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:39:13.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:39:13.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:13.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:39:13.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:13.470+0000 7f4e843cc640 -1 Falling back to public interface 2024-09-10T20:39:13.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:14.713 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:39:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:39:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:14.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:39:15.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:15.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:15.592+0000 7f4e843cc640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:39:16.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:16.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:16.007 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:39:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:39:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:39:16.358 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1332383140,v1:127.0.0.1:6811/1332383140] [v2:127.0.0.1:6812/1332383140,v1:127.0.0.1:6813/1332383140] exists,up 3e3fe996-eb3c-4a7d-9bf5-b91d2b8022a3 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:39:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:16.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:39:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=afdc725a-276f-4a3e-a5c4-c0e6abd34a83 2024-09-10T20:39:16.365 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 afdc725a-276f-4a3e-a5c4-c0e6abd34a83 2024-09-10T20:39:16.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 afdc725a-276f-4a3e-a5c4-c0e6abd34a83' 2024-09-10T20:39:16.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:39:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0ruBmmcB/FhAAsnqlKIWDyNyvjHkGOgC2oQ== 2024-09-10T20:39:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0ruBmmcB/FhAAsnqlKIWDyNyvjHkGOgC2oQ=="}' 2024-09-10T20:39:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new afdc725a-276f-4a3e-a5c4-c0e6abd34a83 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:39:16.731 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:39:16.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:39:16.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQB0ruBmmcB/FhAAsnqlKIWDyNyvjHkGOgC2oQ== --osd-uuid afdc725a-276f-4a3e-a5c4-c0e6abd34a83 2024-09-10T20:39:16.775 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:16.772+0000 7f206e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:16.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:16.812+0000 7f206e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:16.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:16.813+0000 7f206e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:16.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:16.813+0000 7f206e611640 -1 bdev(0x55ce2bdcfc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:39:16.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:16.813+0000 7f206e611640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:39:22.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:39:22.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:39:22.258 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:39:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:39:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:39:22.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:39:22.628 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:39:22.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:22.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:39:22.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:39:22.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:39:22.659 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:22.655+0000 7f44283cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:22.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:22.658+0000 7f44283cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:22.662 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:22.659+0000 7f44283cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:39:22.907 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:39:22.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:22.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:39:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:23.783+0000 7f44283cc640 -1 Falling back to public interface 2024-09-10T20:39:24.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:24.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:24.183 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:39:24.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:39:24.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:24.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:39:24.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:25.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:25.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:25.470 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:39:25.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:39:25.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:25.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:39:25.621 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:25.617+0000 7f44283cc640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:39:25.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:26.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:26.831 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:39:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:39:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:26.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:39:27.131 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4122700095,v1:127.0.0.1:6819/4122700095] [v2:127.0.0.1:6820/4122700095,v1:127.0.0.1:6821/4122700095] exists,up afdc725a-276f-4a3e-a5c4-c0e6abd34a83 2024-09-10T20:39:27.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:39:27.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:39:27.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:39:27.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-09-10T20:39:27.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-10T20:39:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T20:39:27.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:39:28.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-09-10T20:39:29.007 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T20:39:29.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-09-10T20:39:29.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:39:29.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:39:29.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:39:29.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:39:29.019 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:39:29.020 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:39:29.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:39:29.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:39:29.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:39:29.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:39:29.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:39:29.513 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:29.513 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:39:29.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:39:29.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:29.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:39:29.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-10T20:39:29.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-10T20:39:29.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-10T20:39:29.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:29.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:39:29.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705668 2024-09-10T20:39:29.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705668 2024-09-10T20:39:29.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486 1-38654705668' 2024-09-10T20:39:29.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:29.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574850 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574850 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486 1-38654705668 2-55834574850' 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-10T20:39:29.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:29.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:39:29.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-10T20:39:29.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-10T20:39:29.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-10T20:39:29.876 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836486 2024-09-10T20:39:29.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:30.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836486 2024-09-10T20:39:30.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:39:31.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:39:31.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:31.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-10T20:39:31.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:31.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705668 2024-09-10T20:39:31.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:31.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:39:31.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705668 2024-09-10T20:39:31.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:31.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705668 2024-09-10T20:39:31.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2024-09-10T20:39:31.433 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 38654705668 2024-09-10T20:39:31.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:31.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705668 -lt 38654705668 2024-09-10T20:39:31.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:31.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574850 2024-09-10T20:39:31.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:31.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:39:31.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574850 2024-09-10T20:39:31.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:31.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574850 2024-09-10T20:39:31.712 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 55834574850 2024-09-10T20:39:31.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2024-09-10T20:39:31.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:39:31.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574850 -lt 55834574850 2024-09-10T20:39:31.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:39:31.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:39:31.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:39:32.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:39:32.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:39:32.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:39:32.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:39:32.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:39:32.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:39:32.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:39:32.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:39:32.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-09-10T20:39:32.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-09-10T20:39:32.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:39:33.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-10T20:39:33.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:39:33.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:39:33.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:39:33.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:39:33.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:39:33.148 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:33.148 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:39:33.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:39:33.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:33.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:39:33.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:39:33.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:39:33.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:39:33.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:39:33.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:39:33.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:39:34.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:39:34.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T20:39:34.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:39:34.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:39:34.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:39:34.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:39:34.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:39:34.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T20:39:35.438 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T20:39:35.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:35.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:39:35.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:39:35.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:39:35.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:39:35.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:39:35.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:39:35.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:39:35.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:39:35.958 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:39:35.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:39:35.990 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:35.985+0000 7f249afdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:35.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:35.987+0000 7f249afdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:35.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:35.988+0000 7f249afdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:36.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:37.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:37.118+0000 7f249afdd640 -1 Falling back to public interface 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:37.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:38.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:38.976 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:38.973+0000 7f249afdd640 -1 osd.0 18 log_to_monitors true 2024-09-10T20:39:39.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:40.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:40.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:40.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:39:40.167 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:39:40.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:40.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:40.460 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/1814951512,v1:127.0.0.1:6803/1814951512] [v2:127.0.0.1:6804/1814951512,v1:127.0.0.1:6805/1814951512] exists,up da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:39:40.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:39:40.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:39:40.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:39:40.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:39:40.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:39:40.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:39:40.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:39:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:39:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:39:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:39:40.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:40.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:39:40.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280514 2024-09-10T20:39:40.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280514 2024-09-10T20:39:40.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280514' 2024-09-10T20:39:40.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:40.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:39:41.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705671 2024-09-10T20:39:41.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705671 2024-09-10T20:39:41.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280514 1-38654705671' 2024-09-10T20:39:41.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:41.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574853 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574853 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280514 1-38654705671 2-55834574853' 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280514 2024-09-10T20:39:41.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:41.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:39:41.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280514 2024-09-10T20:39:41.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:41.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280514 2024-09-10T20:39:41.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2024-09-10T20:39:41.140 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 94489280514 2024-09-10T20:39:41.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:41.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 94489280514 2024-09-10T20:39:41.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:39:42.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:39:42.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:42.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280514 -lt 94489280514 2024-09-10T20:39:42.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705671 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705671 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705671 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2024-09-10T20:39:42.729 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 38654705671 2024-09-10T20:39:42.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:42.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705672 -lt 38654705671 2024-09-10T20:39:42.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:42.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574853 2024-09-10T20:39:42.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:42.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:39:42.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574853 2024-09-10T20:39:42.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:42.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574853 2024-09-10T20:39:42.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574853' 2024-09-10T20:39:42.977 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 55834574853 2024-09-10T20:39:42.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:39:43.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574854 -lt 55834574853 2024-09-10T20:39:43.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:39:43.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:39:43.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:39:43.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:39:43.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:39:43.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:39:43.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:39:43.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:39:44.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:39:44.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:39:44.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:39:44.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-09-10T20:39:44.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T20:39:44.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:39:44.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:44.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:39:44.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-09-10T20:39:44.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-09-10T20:39:44.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:39:44.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:39:44.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:44.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:39:44.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-09-10T20:39:44.770 INFO:tasks.workunit.client.0.smithi181.stdout:Affected PG 1.0 w/ primary 1 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:39:44.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:39:45.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-09-10T20:39:27.447148+0000 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:39:45.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T20:39:45.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-09-10T20:39:27.447148+0000 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:45.318 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:45.319 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T20:39:45.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-09-10T20:39:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T20:39:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T20:39:45.406 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-09-10T20:39:45.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T20:39:45.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-09-10T20:39:45.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:39:45.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:39:45.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:39:45.762 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:45.763 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:39:45.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:39:45.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:45.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:39:45.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280516 2024-09-10T20:39:45.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280516 2024-09-10T20:39:45.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280516' 2024-09-10T20:39:45.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:45.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:39:45.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705673 2024-09-10T20:39:45.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705673 2024-09-10T20:39:45.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280516 1-38654705673' 2024-09-10T20:39:45.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:45.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:39:46.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574855 2024-09-10T20:39:46.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574855 2024-09-10T20:39:46.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280516 1-38654705673 2-55834574855' 2024-09-10T20:39:46.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:46.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280516 2024-09-10T20:39:46.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:46.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:39:46.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280516 2024-09-10T20:39:46.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:46.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280516 2024-09-10T20:39:46.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280516' 2024-09-10T20:39:46.079 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 94489280516 2024-09-10T20:39:46.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:46.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280516 -lt 94489280516 2024-09-10T20:39:46.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:46.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705673 2024-09-10T20:39:46.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:46.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:39:46.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705673 2024-09-10T20:39:46.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:46.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705673 2024-09-10T20:39:46.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2024-09-10T20:39:46.357 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 38654705673 2024-09-10T20:39:46.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:46.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705673 -lt 38654705673 2024-09-10T20:39:46.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:46.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574855 2024-09-10T20:39:46.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:46.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:39:46.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574855 2024-09-10T20:39:46.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:46.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574855 2024-09-10T20:39:46.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2024-09-10T20:39:46.637 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 55834574855 2024-09-10T20:39:46.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:39:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574855 -lt 55834574855 2024-09-10T20:39:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-09-10T20:39:46.914 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10 20:39:46.911565752+00:00 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T20:39:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-09-10T20:39:47.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-09-10T20:39:47.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:47.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:47.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:47.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:47.110 INFO:tasks.workunit.client.0.smithi181.stdout:{"deep":true,"must":false,"stamp":"2024-08-27T20:38:06.993411+0000"}0 ) state now: "active+clean" 2024-09-10T20:39:47.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-09-10T20:39:47.111 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:47.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:47.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:47.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-10T20:39:47.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:47.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:47.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:47.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:47.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:47.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:47.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-09-10T20:39:47.518 INFO:tasks.workunit.client.0.smithi181.stdout:1 ) state now: "active+clean" 2024-09-10T20:39:47.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:47.519 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:47.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:47.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-10T20:39:47.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:47.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:47.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:47.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:47.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:47.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:47.925 INFO:tasks.workunit.client.0.smithi181.stdout:2 ) state now: "active+clean" 2024-09-10T20:39:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-09-10T20:39:47.926 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:47.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-09-10T20:39:47.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:48.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:48.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:48.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:48.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:48.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:48.332 INFO:tasks.workunit.client.0.smithi181.stdout:3 ) state now: "active+clean" 2024-09-10T20:39:48.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-09-10T20:39:48.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:48.333 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:48.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:48.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-09-10T20:39:48.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:48.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:48.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:48.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:48.741 INFO:tasks.workunit.client.0.smithi181.stdout:4 ) state now: "active+clean" 2024-09-10T20:39:48.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-09-10T20:39:48.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:48.741 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:48.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:48.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-09-10T20:39:48.742 INFO:tasks.workunit.client.0.smithi181.stdout:loop --------> 4 2024-09-10T20:39:48.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-09-10T20:39:48.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:49.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:49.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:49.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:49.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:49.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:49.148 INFO:tasks.workunit.client.0.smithi181.stdout:5 ) state now: "active+clean" 2024-09-10T20:39:49.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-09-10T20:39:49.149 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:49.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:49.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:49.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-09-10T20:39:49.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:49.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:49.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:49.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:49.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:49.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-10T20:39:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-09-10T20:39:49.557 INFO:tasks.workunit.client.0.smithi181.stdout:6 ) state now: "active+clean" 2024-09-10T20:39:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-10T20:39:49.558 INFO:tasks.workunit.client.0.smithi181.stdout:"active+clean" 2024-09-10T20:39:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-09-10T20:39:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-10T20:39:49.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-10T20:39:49.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-10T20:39:49.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-10T20:39:49.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-10T20:39:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:49.964 INFO:tasks.workunit.client.0.smithi181.stdout:7 ) state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-10T20:39:49.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-09-10T20:39:49.965 INFO:tasks.workunit.client.0.smithi181.stdout:found scrub 2024-09-10T20:39:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-09-10T20:39:49.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-09-10T20:39:49.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-09-10T20:39:49.966 INFO:tasks.workunit.client.0.smithi181.stdout:initiate_and_fetch_state ret: 0 2024-09-10T20:39:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-09-10T20:39:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T20:39:49.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.td/osd-scrub-repair.asok 2024-09-10T20:39:49.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-09-10T20:39:50.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2024-09-10T20:39:50.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2024-09-10T20:39:50.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-09-10T20:39:50.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:50.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:50.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.136 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:50.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:50.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-09-10T20:39:50.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-10T20:39:50.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-10T20:39:50.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:50.438 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:50.438 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:50.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.544 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:50.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:50.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-09-10T20:39:50.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-10T20:39:50.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-10T20:39:50.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:50.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:50.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:50.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.953 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:50.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:50.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:50.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-09-10T20:39:50.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-10T20:39:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-10T20:39:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:51.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:51.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:51.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:51.360 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:51.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:51.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:51.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-09-10T20:39:51.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-10T20:39:51.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-10T20:39:51.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:51.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:51.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:51.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-10T20:39:51.769 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:39:51.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:52.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:39:52.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280518 2024-09-10T20:39:52.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280518 2024-09-10T20:39:52.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280518' 2024-09-10T20:39:52.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:52.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:39:52.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705676 2024-09-10T20:39:52.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705676 2024-09-10T20:39:52.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280518 1-38654705676' 2024-09-10T20:39:52.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:39:52.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:39:52.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574858 2024-09-10T20:39:52.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574858 2024-09-10T20:39:52.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280518 1-38654705676 2-55834574858' 2024-09-10T20:39:52.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:52.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280518 2024-09-10T20:39:52.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:52.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:39:52.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280518 2024-09-10T20:39:52.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:52.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280518 2024-09-10T20:39:52.356 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 94489280518 2024-09-10T20:39:52.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280518' 2024-09-10T20:39:52.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:39:52.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280518 -lt 94489280518 2024-09-10T20:39:52.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:52.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705676 2024-09-10T20:39:52.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:52.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:39:52.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705676 2024-09-10T20:39:52.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:52.635 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 38654705676 2024-09-10T20:39:52.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705676 2024-09-10T20:39:52.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2024-09-10T20:39:52.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:52.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705675 -lt 38654705676 2024-09-10T20:39:52.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:39:53.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:39:53.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:54.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705675 -lt 38654705676 2024-09-10T20:39:54.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:39:55.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:39:55.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:39:55.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705676 -lt 38654705676 2024-09-10T20:39:55.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:39:55.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574858 2024-09-10T20:39:55.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:39:55.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:39:55.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574858 2024-09-10T20:39:55.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:39:55.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574858 2024-09-10T20:39:55.488 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 55834574858 2024-09-10T20:39:55.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2024-09-10T20:39:55.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:39:55.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574858 -lt 55834574858 2024-09-10T20:39:55.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-10T20:39:56.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-10T20:39:56.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-10T20:39:56.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-10T20:39:56.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-10T20:39:56.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-09-10T20:39:56.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-09-10T20:39:56.174 INFO:tasks.workunit.client.0.smithi181.stdout:wait-scrub-end state now: "active+clean" 2024-09-10T20:39:56.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:56.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-09-10T20:39:56.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-10T20:39:56.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-09-10T20:39:56.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-09-10T20:39:56.436 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:39:56.436 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-09-10T20:39:49.657211+0000 18'1 23:47 [1,0] 1 [1,0] 1 0'0 2024-08-27T20:38:06.993411+0000 0'0 2024-08-27T20:38:06.993411+0000 0 0 deep scrubbing for 2s 0 0 2024-09-10T20:39:56.437 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T20:39:56.437 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:39:56.437 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:56.449 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:39:56.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:39:56.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:39:56.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:39:57.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:39:57.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:39:57.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:39:57.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:39:57.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:39:57.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:39:57.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:39:57.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-10T20:39:57.777 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:39:57.778 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:39:58.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:39:58.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:39:58.047 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:39:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:39:58.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:39:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:39:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:39:58.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:39:58.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:39:58.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:58.072+0000 7fd358358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:58.079 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:58.074+0000 7fd358358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:58.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:58.076+0000 7fd358358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:39:58.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:39:58.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:39:58.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:39:58.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:58.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:58.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:39:59.193 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:39:59.190+0000 7fd358358640 -1 Falling back to public interface 2024-09-10T20:39:59.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:39:59.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:39:59.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:39:59.646 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:39:59.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:39:59.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:39:59.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:00.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:01.023 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:01.019+0000 7fd358358640 -1 osd.0 23 log_to_monitors true 2024-09-10T20:40:01.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:02.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 27 up_thru 0 down_at 24 last_clean_interval [22,23) [v2:127.0.0.1:6802/3440796453,v1:127.0.0.1:6803/3440796453] [v2:127.0.0.1:6804/3440796453,v1:127.0.0.1:6805/3440796453] exists,up da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:40:02.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:40:02.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:40:02.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:40:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:40:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:40:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:40:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:40:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:02.868 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:40:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964116994 2024-09-10T20:40:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964116994 2024-09-10T20:40:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-115964116994' 2024-09-10T20:40:02.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:02.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:40:03.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705679 2024-09-10T20:40:03.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705679 2024-09-10T20:40:03.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-115964116994 1-38654705679' 2024-09-10T20:40:03.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:03.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:40:03.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574861 2024-09-10T20:40:03.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574861 2024-09-10T20:40:03.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-115964116994 1-38654705679 2-55834574861' 2024-09-10T20:40:03.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:03.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-115964116994 2024-09-10T20:40:03.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:03.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:40:03.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-115964116994 2024-09-10T20:40:03.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:03.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964116994 2024-09-10T20:40:03.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 115964116994' 2024-09-10T20:40:03.179 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 115964116994 2024-09-10T20:40:03.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:40:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 115964116994 2024-09-10T20:40:03.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:40:04.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:40:04.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:40:04.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964116994 -lt 115964116994 2024-09-10T20:40:04.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:04.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705679 2024-09-10T20:40:04.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:04.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:40:04.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705679 2024-09-10T20:40:04.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:04.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705679 2024-09-10T20:40:04.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2024-09-10T20:40:04.746 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 38654705679 2024-09-10T20:40:04.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:40:05.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705679 -lt 38654705679 2024-09-10T20:40:05.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:05.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574861 2024-09-10T20:40:05.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:05.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:40:05.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574861 2024-09-10T20:40:05.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574861 2024-09-10T20:40:05.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574861' 2024-09-10T20:40:05.028 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 55834574861 2024-09-10T20:40:05.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:40:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574861 -lt 55834574861 2024-09-10T20:40:05.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:40:05.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:40:05.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:40:05.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:40:05.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:40:05.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:40:05.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:40:05.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:40:05.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:40:06.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:40:06.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:40:06.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:40:06.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:40:06.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-10T20:40:06.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:40:06.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:40:06.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:40:07.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:40:07.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-10T20:40:07.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:40:07.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:40:07.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:40:07.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:40:07.807 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:40:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:40:07.836 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:07.833+0000 7fd922320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:07.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:07.835+0000 7fd922320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:07.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:07.836+0000 7fd922320640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:08.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:40:08.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:08.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:08.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:08.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:08.956+0000 7fd922320640 -1 Falling back to public interface 2024-09-10T20:40:09.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:09.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:09.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:40:09.381 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:40:09.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:09.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:40:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:40:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:10.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:11.125 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:11.121+0000 7fd922320640 -1 osd.0 28 log_to_monitors true 2024-09-10T20:40:11.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:11.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:40:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:40:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:11.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:12.246 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [27,28) [v2:127.0.0.1:6802/3217435036,v1:127.0.0.1:6803/3217435036] [v2:127.0.0.1:6804/3217435036,v1:127.0.0.1:6805/3217435036] exists,up da301d5e-e0e4-4820-91d0-37a9cf8439e2 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:40:12.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:40:12.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:40:12.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:40:12.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:12.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:40:12.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-10T20:40:12.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-10T20:40:12.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474' 2024-09-10T20:40:12.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:12.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:40:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705682 2024-09-10T20:40:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705682 2024-09-10T20:40:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-38654705682' 2024-09-10T20:40:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:40:12.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:40:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574864 2024-09-10T20:40:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574864 2024-09-10T20:40:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-38654705682 2-55834574864' 2024-09-10T20:40:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:12.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953474 2024-09-10T20:40:12.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:40:12.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953474 2024-09-10T20:40:12.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:12.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-10T20:40:12.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2024-09-10T20:40:12.921 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 137438953474 2024-09-10T20:40:12.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:40:13.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953474 2024-09-10T20:40:13.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:40:14.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:40:14.196 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:40:14.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-10T20:40:14.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:14.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705682 2024-09-10T20:40:14.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:14.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:40:14.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705682 2024-09-10T20:40:14.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705682 2024-09-10T20:40:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2024-09-10T20:40:14.490 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 38654705682 2024-09-10T20:40:14.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:40:14.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705682 -lt 38654705682 2024-09-10T20:40:14.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:40:14.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574864 2024-09-10T20:40:14.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:40:14.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:40:14.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574864 2024-09-10T20:40:14.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:40:14.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574864 2024-09-10T20:40:14.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574864' 2024-09-10T20:40:14.771 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 55834574864 2024-09-10T20:40:14.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:40:15.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574864 -lt 55834574864 2024-09-10T20:40:15.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:40:15.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:40:15.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:40:15.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:40:15.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:40:15.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:40:15.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:40:15.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:40:15.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:40:16.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:40:16.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:40:16.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:40:16.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:40:16.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-10T20:40:16.005 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:39:52.658+0000 7f4e69340640 10 osd.1 pg_epoch: 23 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/23 n=1 ec=15/15 lis/c=22/22 les/c/f=23/23/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-10T20:40:16.006 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:39:52.658+0000 7f4e69340640 10 osd.1 pg_epoch: 23 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/23 n=1 ec=15/15 lis/c=22/22 les/c/f=23/23/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-10T20:40:16.006 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:39:52.658+0000 7f4e69340640 15 osd.1 pg_epoch: 23 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/23 n=1 ec=15/15 lis/c=22/22 les/c/f=23/23/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-10T20:40:16.006 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:39:52.658+0000 7f4e69340640 20 osd.1 pg_epoch: 23 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/23 n=1 ec=15/15 lis/c=22/22 les/c/f=23/23/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-10T20:40:16.006 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T20:39:52.658+0000 7f4e69340640 19 osd.1 pg_epoch: 23 pg[1.0( v 18'1 (0'0,18'1] local-lis/les=22/23 n=1 ec=15/15 lis/c=22/22 les/c/f=23/23/0 sis=22) [1,0] r=0 lpr=22 crt=18'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:40:16.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:40:16.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:40:16.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:40:16.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:40:16.115 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:40:16.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:40:16.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:40:16.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:40:16.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:40:16.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:40:16.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:40:16.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:40:16.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:40:16.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:40:16.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:40:16.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:40:16.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:40:16.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:40:16.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:40:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:40:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:40:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:40:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:40:16.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:40:16.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:40:16.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:40:16.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:40:16.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:40:16.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:40:16.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:40:16.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:40:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:40:16.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:40:16.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:40:16.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:40:16.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:40:16.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:40:16.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:40:16.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:40:16.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:40:16.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:40:16.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:40:16.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:40:16.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:40:16.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:40:16.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:40:16.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:40:16.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:40:16.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:40:16.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:40:16.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:40:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:40:16.586 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.587 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:16.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:40:16.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:40:16.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:40:16.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:40:16.778 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:40:16.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:40:16.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:40:16.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:40:16.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:40:17.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:40:17.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:40:17.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:40:17.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:40:17.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:17.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:17.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:17.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:40:17.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:40:17.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:40:17.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-10T20:40:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:40:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:40:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:40:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:40:17.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:40:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:40:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:40:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:40:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:40:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:40:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:40:17.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:40:17.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:40:17.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:40:17.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:40:17.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b89c67e4-aa74-4739-8f3a-d8781b1e5f7c 2024-09-10T20:40:17.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b89c67e4-aa74-4739-8f3a-d8781b1e5f7c' 2024-09-10T20:40:17.797 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 b89c67e4-aa74-4739-8f3a-d8781b1e5f7c 2024-09-10T20:40:17.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:40:17.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxruBmnFfGLxAAZXz+UnN8Auos3JSGFDLn0w== 2024-09-10T20:40:17.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxruBmnFfGLxAAZXz+UnN8Auos3JSGFDLn0w=="}' 2024-09-10T20:40:17.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b89c67e4-aa74-4739-8f3a-d8781b1e5f7c -i td/osd-scrub-repair/0/new.json 2024-09-10T20:40:18.007 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:40:18.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:40:18.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCxruBmnFfGLxAAZXz+UnN8Auos3JSGFDLn0w== --osd-uuid b89c67e4-aa74-4739-8f3a-d8781b1e5f7c 2024-09-10T20:40:18.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:18.048+0000 7fc947da6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:18.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:18.055+0000 7fc947da6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:18.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:18.056+0000 7fc947da6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:18.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:18.056+0000 7fc947da6640 -1 bdev(0x563083e5dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:40:18.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:18.057+0000 7fc947da6640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:40:23.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:40:23.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:40:23.589 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:40:23.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:40:23.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:40:23.967 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:40:23.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:40:23.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:40:23.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:40:23.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:40:23.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:40:23.998 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:23.993+0000 7f642c028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:24.000 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:23.995+0000 7f642c028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:24.001 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:23.996+0000 7f642c028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:24.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:40:24.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:40:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:24.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:24.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:25.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:25.375+0000 7f642c028640 -1 Falling back to public interface 2024-09-10T20:40:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:40:25.523 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:40:25.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:25.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:25.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:26.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:26.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:26.812 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:40:26.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:40:26.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:26.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:27.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:27.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:27.375+0000 7f642c028640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:40:28.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:28.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:28.096 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:40:28.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:40:28.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:28.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:28.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:29.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:29.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:29.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:40:29.494 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:40:29.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:29.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:29.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:30.782 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:40:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:40:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:31.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:32.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:32.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:32.165 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:40:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:40:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:32.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:32.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:33.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:33.454 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:40:33.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:33.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:40:33.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:33.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:33.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:34.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:34.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:34.745 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:40:34.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:40:34.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:34.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:35.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:36.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:36.125 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:40:36.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:40:36.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:36.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:36.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:37.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:37.415 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:40:37.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:40:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:37.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:37.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:38.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:38.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:38.703 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:40:38.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:40:38.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:38.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:39.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:40.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:40.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:40.244 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:40:40.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:40:40.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:40.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:40.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:41.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:41.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:41.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:40:41.709 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:40:41.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:41.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:42.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:43.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:43.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:43.433 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:40:43.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:40:43.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:43.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:43.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:44.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:44.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:44.984 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:40:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:40:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:45.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:46.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:46.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:46.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:40:46.639 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:40:46.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:46.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:47.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:48.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:48.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:48.022 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:40:48.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:40:48.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:48.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:48.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:49.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:49.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:49.642 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:40:49.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:40:49.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:49.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:40:49.931 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3875905831,v1:127.0.0.1:6803/3875905831] [v2:127.0.0.1:6804/3875905831,v1:127.0.0.1:6805/3875905831] exists,up b89c67e4-aa74-4739-8f3a-d8781b1e5f7c 2024-09-10T20:40:49.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:40:49.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:40:49.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:40:49.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:40:49.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:40:49.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:40:49.938 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:40:49.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:40:49.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6c83d314-76a7-4b6b-b9e3-7029960b8c52' 2024-09-10T20:40:49.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:40:49.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRruBmwBqWOBAAtQ9bey5q1BOanrFbpx3M0w== 2024-09-10T20:40:49.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRruBmwBqWOBAAtQ9bey5q1BOanrFbpx3M0w=="}' 2024-09-10T20:40:49.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c83d314-76a7-4b6b-b9e3-7029960b8c52 -i td/osd-scrub-repair/1/new.json 2024-09-10T20:40:50.276 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:40:50.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:40:50.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDRruBmwBqWOBAAtQ9bey5q1BOanrFbpx3M0w== --osd-uuid 6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:40:50.321 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:50.317+0000 7ff2b0644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:50.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:50.357+0000 7ff2b0644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:50.361 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:50.358+0000 7ff2b0644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:50.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:50.358+0000 7ff2b0644640 -1 bdev(0x5582425b7c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:40:50.362 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:50.358+0000 7ff2b0644640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:40:55.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:40:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:40:55.821 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:40:55.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:40:55.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:40:56.199 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:40:56.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:40:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:40:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:40:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:40:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:40:56.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:56.227+0000 7f3268a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:56.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:56.229+0000 7f3268a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:56.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:56.230+0000 7f3268a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:40:56.477 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:40:56.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:56.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:40:56.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:57.588 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:57.584+0000 7f3268a0f640 -1 Falling back to public interface 2024-09-10T20:40:57.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:57.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:57.754 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:40:57.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:40:57.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:57.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:40:58.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:40:59.043 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:40:59.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:40:59.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:40:59.406+0000 7f3268a0f640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:41:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:00.334 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:41:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:41:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:00.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:01.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:01.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:01.654 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:41:01.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:41:01.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:01.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:01.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:02.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:02.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:02.990 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:41:02.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:41:02.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:02.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:03.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:04.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:04.301 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:41:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:41:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:04.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:05.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:05.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:05.713 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:41:05.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:41:05.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:05.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:07.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:07.031 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:41:07.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:41:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:07.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:07.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:08.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:08.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:08.321 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:41:08.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:41:08.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:08.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:08.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:09.666 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:41:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:41:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:09.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:09.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:10.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:10.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:10.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:41:10.947 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:41:10.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:11.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:12.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:12.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:12.348 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:41:12.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:41:12.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:12.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:12.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:13.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:13.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:13.787 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:41:13.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:41:13.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:13.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:14.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:15.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:15.739 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:41:15.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:15.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:41:15.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:15.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:16.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:17.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:17.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:17.256 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:41:17.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:41:17.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:17.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:18.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:19.243 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:41:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:41:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:19.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:19.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:20.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:20.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:20.828 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:41:20.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:41:20.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:20.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:21.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:22.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:22.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:22.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:41:22.529 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:41:22.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:22.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:22.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:23.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:23.863 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:41:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:41:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:23.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:41:24.141 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2599686387,v1:127.0.0.1:6811/2599686387] [v2:127.0.0.1:6812/2599686387,v1:127.0.0.1:6813/2599686387] exists,up 6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:41:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:41:24.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:41:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a482791e-280f-4517-8d7b-94c548505882 2024-09-10T20:41:24.148 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 a482791e-280f-4517-8d7b-94c548505882 2024-09-10T20:41:24.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a482791e-280f-4517-8d7b-94c548505882' 2024-09-10T20:41:24.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:41:24.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0ruBmMSeGCRAA5qhB5XEPX3TMntwDVf0OWw== 2024-09-10T20:41:24.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0ruBmMSeGCRAA5qhB5XEPX3TMntwDVf0OWw=="}' 2024-09-10T20:41:24.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a482791e-280f-4517-8d7b-94c548505882 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:41:24.507 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:41:24.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:41:24.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD0ruBmMSeGCRAA5qhB5XEPX3TMntwDVf0OWw== --osd-uuid a482791e-280f-4517-8d7b-94c548505882 2024-09-10T20:41:24.554 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:24.550+0000 7f81eca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:24.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:24.588+0000 7f81eca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:24.589+0000 7f81eca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:24.590+0000 7f81eca0d640 -1 bdev(0x55d8468b7c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:41:24.593 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:24.590+0000 7f81eca0d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:41:30.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:41:30.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:41:30.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:41:30.877 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:41:30.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:41:31.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:41:31.235 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:41:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:41:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:41:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:41:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:41:31.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:31.262+0000 7fbb2080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:31.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:31.264+0000 7fbb2080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:31.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:31.265+0000 7fbb2080d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:41:31.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:41:31.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:41:31.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:41:31.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:41:31.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:41:31.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:31.512 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:41:31.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:41:31.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:31.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:31.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:32.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:32.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:32.797 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:41:32.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:41:32.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:32.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:33.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:33.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:33.112+0000 7fbb2080d640 -1 Falling back to public interface 2024-09-10T20:41:34.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:34.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:34.096 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:41:34.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:41:34.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:34.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:34.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:41:34.777+0000 7fbb2080d640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:41:35.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:35.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:35.376 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:41:35.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:41:35.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:35.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:36.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:36.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:36.679 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:41:36.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:41:36.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:36.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:36.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:37.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:37.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:37.968 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:41:37.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:41:37.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:37.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:38.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:39.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:39.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:39.317 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:41:39.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:41:39.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:39.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:39.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:40.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:40.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:40.686 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:41:40.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:41:40.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:40.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:40.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:41.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:41.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:41.976 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:41:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:41:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:42.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:43.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:43.326 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:41:43.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:41:43.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:43.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:43.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:44.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:44.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:44.630 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:41:44.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:41:44.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:44.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:45.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:46.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:46.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:46.101 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:41:46.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:41:46.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:46.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:46.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:47.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:47.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:47.645 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:41:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:41:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:48.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:49.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:49.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:49.189 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:41:49.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:41:49.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:49.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:49.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:50.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:50.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:50.737 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:41:50.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:41:50.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:50.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:51.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:52.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:52.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:52.550 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:41:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:41:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:52.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:52.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:53.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:53.872 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:41:53.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:41:53.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:53.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:55.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:55.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:55.834 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:41:55.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:41:55.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:55.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:56.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:57.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:57.681 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:41:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:41:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:57.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:58.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:41:59.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:41:59.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:41:59.076 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:41:59.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:41:59.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:41:59.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:41:59.405 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1784475190,v1:127.0.0.1:6819/1784475190] [v2:127.0.0.1:6820/1784475190,v1:127.0.0.1:6821/1784475190] exists,up a482791e-280f-4517-8d7b-94c548505882 2024-09-10T20:41:59.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:41:59.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:41:59.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:41:59.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-10T20:41:59.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T20:41:59.719 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T20:41:59.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T20:41:59.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T20:42:00.212 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T20:42:00.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:42:01.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T20:42:03.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-10T20:42:03.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:42:03.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:42:03.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:42:03.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:42:03.411 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:42:03.411 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:42:03.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:42:03.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:42:03.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:42:03.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:42:03.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:42:03.829 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:03.829 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:42:03.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:42:03.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:03.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:42:03.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T20:42:03.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T20:42:03.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T20:42:03.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:03.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:42:04.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476753 2024-09-10T20:42:04.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476753 2024-09-10T20:42:04.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476753' 2024-09-10T20:42:04.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:04.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476753 2-141733920772' 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T20:42:04.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:04.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:42:04.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T20:42:04.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:04.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T20:42:04.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T20:42:04.141 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T20:42:04.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:04.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T20:42:04.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:05.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:42:05.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:05.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T20:42:05.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:06.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T20:42:06.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-10T20:42:07.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:07.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476753 2024-09-10T20:42:07.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:07.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:42:07.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476753 2024-09-10T20:42:07.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:07.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476753 2024-09-10T20:42:07.037 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 68719476753 2024-09-10T20:42:07.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476753' 2024-09-10T20:42:07.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:42:07.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476753 -lt 68719476753 2024-09-10T20:42:07.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:07.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920772 2024-09-10T20:42:07.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:07.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:42:07.318 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920772 2024-09-10T20:42:07.318 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:07.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-10T20:42:07.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-09-10T20:42:07.320 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920772 2024-09-10T20:42:07.320 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:42:07.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920773 -lt 141733920772 2024-09-10T20:42:07.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:42:07.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:07.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:42:08.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:42:08.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:42:08.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:42:08.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:42:08.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:42:08.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:08.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T20:42:08.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T20:42:08.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T20:42:08.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T20:42:08.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T20:42:09.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T20:42:09.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T20:42:09.745 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T20:42:09.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:42:10.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:42:10.759 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:42:10.760 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:42:10.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:42:10.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:42:10.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:42:10.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:42:10.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:42:10.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:42:10.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:42:10.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:42:10.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:11.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:42:11.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-10T20:42:11.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-10T20:42:11.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-10T20:42:11.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:11.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:42:11.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476755 2024-09-10T20:42:11.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476755 2024-09-10T20:42:11.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-68719476755' 2024-09-10T20:42:11.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:11.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920775 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920775 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-68719476755 2-141733920775' 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-10T20:42:11.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:11.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:42:11.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-10T20:42:11.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:11.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-10T20:42:11.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-10T20:42:11.507 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836504 2024-09-10T20:42:11.507 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:11.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-10T20:42:11.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:12.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:42:12.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-10T20:42:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:13.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476755 2024-09-10T20:42:13.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:42:13.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476755 2024-09-10T20:42:13.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:13.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476755 2024-09-10T20:42:13.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476755' 2024-09-10T20:42:13.076 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 68719476755 2024-09-10T20:42:13.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:42:13.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476755 -lt 68719476755 2024-09-10T20:42:13.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:13.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920775 2024-09-10T20:42:13.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:13.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:42:13.352 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920775 2024-09-10T20:42:13.352 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:13.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920775 2024-09-10T20:42:13.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920775' 2024-09-10T20:42:13.353 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920775 2024-09-10T20:42:13.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:42:13.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920775 -lt 141733920775 2024-09-10T20:42:13.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:42:13.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:13.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:42:13.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:42:13.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:42:14.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:42:14.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:42:14.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:14.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-10T20:42:14.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:42:14.958 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:42:15.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-10T20:42:15.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:42:15.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-10T20:42:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:42:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:42:15.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:42:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:42:15.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:42:15.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:42:17.554 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T20:42:17.554 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-10T20:42:17.554 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:42:18.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:42:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:42:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:18.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:42:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:42:18.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:42:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:42:18.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:42:18.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:18.107+0000 7fd94909c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:18.114 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:18.110+0000 7fd94909c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:18.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:18.112+0000 7fd94909c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:18.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:18.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:19.233+0000 7fd94909c640 -1 Falling back to public interface 2024-09-10T20:42:19.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:19.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:19.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:42:19.640 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:19.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:19.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:20.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:20.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:20.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:42:20.954 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:42:20.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:20.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:21.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:21.254 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:21.251+0000 7fd94909c640 -1 osd.1 56 log_to_monitors true 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/3730016896,v1:127.0.0.1:6811/3730016896] [v2:127.0.0.1:6812/3730016896,v1:127.0.0.1:6813/3730016896] exists,up 6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:42:22.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:42:22.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:42:22.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:42:22.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:42:22.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:42:22.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:42:22.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:42:22.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:23.115 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:42:23.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-10T20:42:23.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-10T20:42:23.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-10T20:42:23.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:23.219 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:42:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972354 2024-09-10T20:42:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972354 2024-09-10T20:42:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-266287972354' 2024-09-10T20:42:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:23.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:42:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920778 2024-09-10T20:42:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920778 2024-09-10T20:42:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-266287972354 2-141733920778' 2024-09-10T20:42:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:23.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-10T20:42:23.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:42:23.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-10T20:42:23.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-10T20:42:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-10T20:42:23.448 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836507 2024-09-10T20:42:23.448 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:23.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-10T20:42:23.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:24.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:42:24.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:25.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-10T20:42:25.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:25.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-266287972354 2024-09-10T20:42:25.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:25.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:42:25.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-266287972354 2024-09-10T20:42:25.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:25.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972354 2024-09-10T20:42:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 266287972354' 2024-09-10T20:42:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 266287972354 2024-09-10T20:42:25.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:42:25.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972354 -lt 266287972354 2024-09-10T20:42:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:25.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920778 2024-09-10T20:42:25.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:25.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:42:25.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920778 2024-09-10T20:42:25.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920778 2024-09-10T20:42:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920778' 2024-09-10T20:42:25.287 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 141733920778 2024-09-10T20:42:25.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:42:25.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920778 -lt 141733920778 2024-09-10T20:42:25.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:42:25.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:25.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:42:25.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:42:26.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:42:26.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:42:26.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:26.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:26.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:42:26.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:42:26.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:42:26.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-10T20:42:26.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-10T20:42:26.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:42:26.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:42:26.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:42:26.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-10T20:42:09.573966+0000 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:42:09.573966+0000 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:42:27.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:42:27.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:42:09.573966+0000 '>' 2024-09-10T20:42:09.573966+0000 2024-09-10T20:42:27.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:42:28.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:42:28.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:42:28.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:42:28.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:42:28.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:42:28.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:42:28.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:42:28.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:42:09.573966+0000 '>' 2024-09-10T20:42:09.573966+0000 2024-09-10T20:42:28.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:42:29.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:42:29.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:42:29.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:42:29.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:42:29.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:42:29.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:42:29.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:42:30.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:42:26.511957+0000 '>' 2024-09-10T20:42:09.573966+0000 2024-09-10T20:42:30.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:42:30.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-10T20:42:30.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:42:30.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:42:30.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:42:30.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:42:30.003 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:42:30.003 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:42:30.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:42:30.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:42:30.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:42:30.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:42:30.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:42:30.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:42:30.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:42:30.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:42:30.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:30.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:42:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-10T20:42:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-10T20:42:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-10T20:42:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:30.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:42:30.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972356 2024-09-10T20:42:30.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972356 2024-09-10T20:42:30.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-266287972356' 2024-09-10T20:42:30.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:30.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:42:30.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920781 2024-09-10T20:42:30.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920781 2024-09-10T20:42:30.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-266287972356 2-141733920781' 2024-09-10T20:42:30.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:30.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-10T20:42:30.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:30.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:42:30.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-10T20:42:30.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:30.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-10T20:42:30.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-10T20:42:30.710 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836510 2024-09-10T20:42:30.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:30.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-10T20:42:30.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:42:31.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:32.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-10T20:42:32.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:32.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-266287972356 2024-09-10T20:42:32.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:32.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:42:32.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-266287972356 2024-09-10T20:42:32.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:32.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972356 2024-09-10T20:42:32.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 266287972356' 2024-09-10T20:42:32.270 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 266287972356 2024-09-10T20:42:32.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:42:32.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972357 -lt 266287972356 2024-09-10T20:42:32.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:32.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920781 2024-09-10T20:42:32.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:42:32.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920781 2024-09-10T20:42:32.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920781 2024-09-10T20:42:32.553 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920781 2024-09-10T20:42:32.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920781' 2024-09-10T20:42:32.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:42:32.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920781 -lt 141733920781 2024-09-10T20:42:32.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:42:32.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:32.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:33.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:42:33.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:42:33.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:42:33.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:42:33.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:33.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:42:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-10T20:42:33.810 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:42:33.810 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:42:33.810 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:42:33.810 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:42:34.086 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-10T20:42:34.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:42:34.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:42:34.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:42:34.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:42:34.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:42:34.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:42:34.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:42:34.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:42:34.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:42:34.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:42:34.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:42:34.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:42:34.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:42:34.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:42:34.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:42:34.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:42:35.156 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T20:42:35.156 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-10T20:42:35.156 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:42:35.156 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:42:35.156 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:42:35.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:35.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:42:35.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:42:35.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:42:35.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:35.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:42:35.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:42:35.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:42:35.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:42:35.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:42:35.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:35.450+0000 7f34b791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:35.551+0000 7f34b791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:35.552+0000 7f34b791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:35.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:42:35.762 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:42:35.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:35.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:36.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:36.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:36.656+0000 7f34b791b640 -1 Falling back to public interface 2024-09-10T20:42:37.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:37.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:37.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:42:37.041 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:37.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:37.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:37.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:38.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:42:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:42:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:38.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:38.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:38.835 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:38.831+0000 7f34b791b640 -1 osd.1 71 log_to_monitors true 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:39.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 72 last_clean_interval [62,71) [v2:127.0.0.1:6810/4004610086,v1:127.0.0.1:6811/4004610086] [v2:127.0.0.1:6812/4004610086,v1:127.0.0.1:6813/4004610086] exists,up 6c83d314-76a7-4b6b-b9e3-7029960b8c52 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:42:39.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:42:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:42:39.938 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:42:39.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:42:39.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:42:39.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:42:40.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:42:40.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:42:40.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:42:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:42:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:42:40.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:40.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:42:40.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-10T20:42:40.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-10T20:42:40.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-10T20:42:40.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:40.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:42:40.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-10T20:42:40.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-10T20:42:40.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-335007449090' 2024-09-10T20:42:40.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:42:40.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:42:40.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920784 2024-09-10T20:42:40.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920784 2024-09-10T20:42:40.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-335007449090 2-141733920784' 2024-09-10T20:42:40.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:40.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-10T20:42:40.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:40.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:42:40.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-10T20:42:40.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:40.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-10T20:42:40.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-10T20:42:40.780 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836513 2024-09-10T20:42:40.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-10T20:42:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:42:42.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:42:42.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:42:42.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-10T20:42:42.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:42.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-335007449090 2024-09-10T20:42:42.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:42.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:42:42.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-335007449090 2024-09-10T20:42:42.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:42.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-10T20:42:42.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-09-10T20:42:42.349 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 335007449090 2024-09-10T20:42:42.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:42:42.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449090 -lt 335007449090 2024-09-10T20:42:42.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:42:42.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920784 2024-09-10T20:42:42.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:42:42.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:42:42.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920784 2024-09-10T20:42:42.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:42:42.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920784 2024-09-10T20:42:42.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920784' 2024-09-10T20:42:42.645 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 141733920784 2024-09-10T20:42:42.646 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:42:42.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920784 -lt 141733920784 2024-09-10T20:42:42.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:42:42.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:42.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:43.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:42:43.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:42:43.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:42:43.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:42:43.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:42:43.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:42:43.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:42:43.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:42:43.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:42:43.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:42:43.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:42:43.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:42:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:42:43.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:42:43.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:42:44.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:42:44.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:42:44.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:42:44.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:42:44.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:42:44.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:42:44.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:42:44.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:42:44.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:42:44.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:42:44.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:42:44.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:42:44.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:42:44.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:42:44.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:42:44.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:42:44.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:42:44.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:42:44.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:42:44.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:42:44.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:42:44.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:42:44.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:42:44.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:42:44.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:42:44.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:42:44.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:42:44.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:42:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:42:44.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:42:44.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:42:44.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:42:44.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:42:44.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:42:44.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:42:44.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:42:44.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:42:44.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:42:44.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:42:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:42:44.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:42:44.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:42:44.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:42:44.774 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:42:44.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:42:44.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:42:44.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:42:44.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:42:44.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:42:44.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:42:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:44.859 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:42:44.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:42:44.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:42:44.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T20:42:44.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:42:44.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:42:44.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:42:44.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:42:44.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:42:44.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:42:45.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:42:45.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:42:45.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:42:45.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:42:45.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:42:45.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:42:45.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:42:45.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:42:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:42:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:42:45.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:42:45.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c77264b9-24c7-416e-becb-bf1e52d62124 2024-09-10T20:42:45.768 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 c77264b9-24c7-416e-becb-bf1e52d62124 2024-09-10T20:42:45.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c77264b9-24c7-416e-becb-bf1e52d62124' 2024-09-10T20:42:45.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:42:45.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFr+BmcTiDLhAAsjCBr9Lqd5dPZu4ig/a9BQ== 2024-09-10T20:42:45.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFr+BmcTiDLhAAsjCBr9Lqd5dPZu4ig/a9BQ=="}' 2024-09-10T20:42:45.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c77264b9-24c7-416e-becb-bf1e52d62124 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:42:45.971 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:42:45.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:42:45.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBFr+BmcTiDLhAAsjCBr9Lqd5dPZu4ig/a9BQ== --osd-uuid c77264b9-24c7-416e-becb-bf1e52d62124 2024-09-10T20:42:46.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:46.013+0000 7f8be5ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:46.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:46.015+0000 7f8be5ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:46.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:46.016+0000 7f8be5ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:46.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:46.017+0000 7f8be5ba2640 -1 bdev(0x5605026b7c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:42:46.020 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:46.017+0000 7f8be5ba2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:42:51.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:42:51.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:42:51.445 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:42:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:42:51.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:42:51.805 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:42:51.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:42:51.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:42:51.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:42:51.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:42:51.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:42:51.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:51.833+0000 7f9f4342f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:51.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:51.835+0000 7f9f4342f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:51.840 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:51.836+0000 7f9f4342f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:42:52.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:42:52.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:42:52.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:42:52.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:42:52.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:42:52.081 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:42:52.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:52.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:42:52.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:52.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:52.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:52.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:52.934+0000 7f9f4342f640 -1 Falling back to public interface 2024-09-10T20:42:53.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:53.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:53.354 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:42:53.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:42:53.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:53.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:54.510 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:42:54.505+0000 7f9f4342f640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:42:54.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:54.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:54.647 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:42:54.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:42:54.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:54.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:54.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:55.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:55.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:55.946 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:42:55.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:42:55.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:55.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:56.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:57.236 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:42:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:42:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:57.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:58.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:58.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:58.521 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:42:58.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:42:58.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:58.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:42:58.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:42:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:42:59.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:42:59.813 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:42:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:42:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:42:59.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:00.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:01.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:01.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:01.226 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:43:01.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:43:01.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:01.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:01.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:02.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:02.515 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:43:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:43:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:02.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:02.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:03.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:03.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:03.808 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:43:03.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:43:03.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:03.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:04.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:05.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:05.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:05.097 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:43:05.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:43:05.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:05.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:05.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:06.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:06.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:06.451 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:43:06.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:43:06.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:06.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:06.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:07.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:07.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:07.894 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:43:07.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:43:07.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:07.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:08.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:09.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:09.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:09.570 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:43:09.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:43:09.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:09.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:10.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:11.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:11.471 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:43:11.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:11.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:43:11.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:11.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:11.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:12.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:12.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:12.908 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:43:12.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:43:12.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:12.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:13.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:14.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:14.627 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:43:14.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:14.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:43:14.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:14.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:14.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:15.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:15.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:15.907 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:43:15.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:43:15.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:15.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:16.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:17.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:17.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:17.582 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:43:17.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:43:17.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:17.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:43:17.881 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4185273309,v1:127.0.0.1:6803/4185273309] [v2:127.0.0.1:6804/4185273309,v1:127.0.0.1:6805/4185273309] exists,up c77264b9-24c7-416e-becb-bf1e52d62124 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:43:17.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:43:17.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:43:17.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2ab9a24e-df6e-4029-91e9-023b658b7dad' 2024-09-10T20:43:17.888 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:43:17.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:43:17.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlr+BmVy63NRAAmy9bovhaT7xOW9q02djeJw== 2024-09-10T20:43:17.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlr+BmVy63NRAAmy9bovhaT7xOW9q02djeJw=="}' 2024-09-10T20:43:17.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2ab9a24e-df6e-4029-91e9-023b658b7dad -i td/osd-scrub-repair/1/new.json 2024-09-10T20:43:18.238 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:43:18.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:43:18.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBlr+BmVy63NRAAmy9bovhaT7xOW9q02djeJw== --osd-uuid 2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:43:18.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:18.279+0000 7f3ab2156640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:18.322 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:18.319+0000 7f3ab2156640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:18.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:18.320+0000 7f3ab2156640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:18.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:18.320+0000 7f3ab2156640 -1 bdev(0x5558098a3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:43:18.324 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:18.320+0000 7f3ab2156640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:43:24.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:43:24.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:43:24.697 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:43:24.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:43:24.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:43:25.059 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:43:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:43:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:43:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:43:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:43:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:43:25.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:25.087+0000 7f588fce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:25.092 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:25.089+0000 7f588fce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:25.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:25.090+0000 7f588fce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:43:25.334 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:43:25.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:25.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:25.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:26.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:26.460+0000 7f588fce4640 -1 Falling back to public interface 2024-09-10T20:43:26.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:26.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:26.612 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:43:26.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:43:26.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:26.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:26.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:27.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:27.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:27.901 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:43:27.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:43:27.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:27.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:28.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:28.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:28.478+0000 7f588fce4640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:43:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:29.196 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:43:29.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:43:29.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:29.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:29.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:30.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:30.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:30.488 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:43:30.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:43:30.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:30.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:30.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:31.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:31.843 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:43:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:43:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:32.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:33.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:33.216 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:43:33.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:33.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:43:33.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:33.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:33.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:34.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:34.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:34.508 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:43:34.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:43:34.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:34.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:34.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:35.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:35.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:35.841 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:43:35.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:43:35.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:35.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:36.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:37.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:37.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:37.153 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:43:37.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:43:37.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:37.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:38.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:38.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:38.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:43:38.448 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:43:38.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:38.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:39.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:39.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:39.816 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:43:39.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:43:39.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:39.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:40.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:41.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:41.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:41.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:43:41.302 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:43:41.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:41.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:41.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:42.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:42.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:42.994 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:43:42.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:43:42.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:42.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:43.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:44.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:44.839 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:43:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:43:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:44.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:45.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:46.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:46.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:46.326 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:43:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:43:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:46.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:47.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:47.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:47.783 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:43:47.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:43:47.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:47.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:48.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:43:49.072 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:43:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:49.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:50.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:50.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:50.601 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:43:50.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:43:50.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:50.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:51.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:43:52.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:43:52.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:52.094 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:43:52.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:43:52.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:52.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:43:52.377 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/213620753,v1:127.0.0.1:6811/213620753] [v2:127.0.0.1:6812/213620753,v1:127.0.0.1:6813/213620753] exists,up 2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:43:52.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:43:52.383 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:43:52.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=53ffb35d-94b1-41ba-b209-49e65840f2dd 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 53ffb35d-94b1-41ba-b209-49e65840f2dd' 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 53ffb35d-94b1-41ba-b209-49e65840f2dd 2024-09-10T20:43:52.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:43:52.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCIr+BmQbWcFxAAJFDoVGnXd/HywmqQrj9jSg== 2024-09-10T20:43:52.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCIr+BmQbWcFxAAJFDoVGnXd/HywmqQrj9jSg=="}' 2024-09-10T20:43:52.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 53ffb35d-94b1-41ba-b209-49e65840f2dd -i td/osd-scrub-repair/2/new.json 2024-09-10T20:43:52.761 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:43:52.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:43:52.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCIr+BmQbWcFxAAJFDoVGnXd/HywmqQrj9jSg== --osd-uuid 53ffb35d-94b1-41ba-b209-49e65840f2dd 2024-09-10T20:43:52.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:52.802+0000 7fe7a7418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:52.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:52.833+0000 7fe7a7418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:52.838 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:52.834+0000 7fe7a7418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:52.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:52.834+0000 7fe7a7418640 -1 bdev(0x55ff4fe45c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:43:52.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:52.834+0000 7fe7a7418640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:43:59.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:43:59.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:43:59.282 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:43:59.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:43:59.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:43:59.639 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:43:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:43:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:43:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:43:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:43:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:43:59.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:59.667+0000 7f4c91a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:59.673 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:59.669+0000 7f4c91a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:59.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:43:59.670+0000 7f4c91a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:43:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:43:59.929 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:43:59.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:43:59.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:43:59.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:00.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:01.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:01.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:01.208 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:44:01.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:44:01.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:01.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:01.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:01.532 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:01.529+0000 7f4c91a2e640 -1 Falling back to public interface 2024-09-10T20:44:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:02.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:02.505 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:44:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:44:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:02.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:03.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:03.382+0000 7f4c91a2e640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:44:03.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:03.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:03.792 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:44:03.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:44:03.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:03.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:04.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:05.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:05.301 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:44:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:44:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:05.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:05.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:06.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:06.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:06.589 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:44:06.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:44:06.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:06.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:06.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:07.874 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:44:07.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:44:07.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:07.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:08.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:09.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:09.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:09.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:44:09.297 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:44:09.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:09.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:09.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:10.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:10.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:10.586 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:44:10.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:44:10.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:10.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:10.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:11.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:11.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:11.874 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:44:11.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:44:11.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:11.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:12.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:13.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:13.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:13.338 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:44:13.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:44:13.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:13.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:13.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:14.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:14.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:14.668 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:44:14.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:44:14.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:14.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:14.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:15.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:15.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:15.959 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:44:15.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:44:15.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:15.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:16.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:17.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:17.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:17.360 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:44:17.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:44:17.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:17.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:18.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:18.713 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:44:18.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:44:18.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:18.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:19.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:20.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:20.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:20.242 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:44:20.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:44:20.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:20.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:20.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:21.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:21.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:21.529 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:44:21.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:44:21.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:21.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:22.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:23.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:23.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:23.238 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:44:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:44:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:23.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:23.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:24.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:24.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:24.800 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:44:24.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:44:24.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:24.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:25.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:26.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:26.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:26.407 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:44:26.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:44:26.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:26.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:26.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:27.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:27.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:27.775 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T20:44:27.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T20:44:27.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:27.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:44:28.075 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2903571041,v1:127.0.0.1:6819/2903571041] [v2:127.0.0.1:6820/2903571041,v1:127.0.0.1:6821/2903571041] exists,up 53ffb35d-94b1-41ba-b209-49e65840f2dd 2024-09-10T20:44:28.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:44:28.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:44:28.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:44:28.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-10T20:44:28.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T20:44:28.343 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T20:44:28.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T20:44:28.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T20:44:28.818 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T20:44:28.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:44:29.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T20:44:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-10T20:44:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:44:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:44:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:44:31.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:44:31.996 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:44:31.996 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:44:31.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:44:31.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:44:31.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:44:32.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:44:32.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:44:32.372 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:44:32.372 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:44:32.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:44:32.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:32.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:44:32.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-10T20:44:32.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-10T20:44:32.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-10T20:44:32.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:32.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:44:32.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476748 2024-09-10T20:44:32.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476748 2024-09-10T20:44:32.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-68719476748' 2024-09-10T20:44:32.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:32.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:44:32.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920774 2024-09-10T20:44:32.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920774 2024-09-10T20:44:32.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-68719476748 2-141733920774' 2024-09-10T20:44:32.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:32.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-10T20:44:32.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:44:32.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-10T20:44:32.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-10T20:44:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-10T20:44:32.686 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836499 2024-09-10T20:44:32.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:32.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-10T20:44:32.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:44:33.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:44:33.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-10T20:44:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:34.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476748 2024-09-10T20:44:34.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:34.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:44:34.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476748 2024-09-10T20:44:34.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:34.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476748 2024-09-10T20:44:34.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476748' 2024-09-10T20:44:34.244 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 68719476748 2024-09-10T20:44:34.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:44:34.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476748 -lt 68719476748 2024-09-10T20:44:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:34.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920774 2024-09-10T20:44:34.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:34.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:44:34.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920774 2024-09-10T20:44:34.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:34.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920774 2024-09-10T20:44:34.525 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920774 2024-09-10T20:44:34.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920774' 2024-09-10T20:44:34.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:44:34.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920775 -lt 141733920774 2024-09-10T20:44:34.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:44:34.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:34.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:35.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:35.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:35.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:35.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:44:35.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:35.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:35.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T20:44:35.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T20:44:35.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T20:44:35.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T20:44:35.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T20:44:36.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T20:44:36.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T20:44:37.291 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T20:44:37.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:44:38.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T20:44:38.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T20:44:38.739 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 2 allow_ec_overwrites to true 2024-09-10T20:44:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T20:44:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:44:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:44:38.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:44:38.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:44:38.752 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:44:38.752 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:44:38.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:44:38.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:44:38.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:44:38.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:44:38.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:44:38.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:44:38.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:44:38.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:44:38.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:44:39.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:44:39.117 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:44:39.117 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:44:39.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:44:39.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:39.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:44:39.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T20:44:39.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T20:44:39.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T20:44:39.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:39.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:44:39.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476750 2024-09-10T20:44:39.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476750 2024-09-10T20:44:39.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476750' 2024-09-10T20:44:39.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:39.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:44:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920777 2024-09-10T20:44:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920777 2024-09-10T20:44:39.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476750 2-141733920777' 2024-09-10T20:44:39.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:39.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T20:44:39.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:39.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:44:39.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T20:44:39.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:39.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T20:44:39.465 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T20:44:39.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T20:44:39.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:39.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T20:44:39.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:44:40.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:44:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:41.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-10T20:44:41.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:41.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476750 2024-09-10T20:44:41.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:41.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:44:41.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476750 2024-09-10T20:44:41.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:41.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476750 2024-09-10T20:44:41.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-09-10T20:44:41.033 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 68719476750 2024-09-10T20:44:41.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:44:41.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476750 -lt 68719476750 2024-09-10T20:44:41.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:41.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920777 2024-09-10T20:44:41.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:41.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:44:41.310 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920777 2024-09-10T20:44:41.311 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:41.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920777 2024-09-10T20:44:41.312 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920777 2024-09-10T20:44:41.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920777' 2024-09-10T20:44:41.312 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:44:41.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920777 -lt 141733920777 2024-09-10T20:44:41.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:44:41.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:41.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:41.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:41.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:42.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:44:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:42.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-10T20:44:42.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:44:42.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-10T20:44:42.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-10T20:44:42.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:44:42.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-10T20:44:42.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:44:42.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:44:42.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:44:42.844 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:44:43.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-10T20:44:43.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:44:43.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:44:43.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:44:43.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:44:44.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:44:44.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:44:44.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:44:44.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:44:44.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:44:44.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:44:44.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:44:44.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:44:46.295 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T20:44:46.295 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-10T20:44:46.295 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:44:46.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:44:46.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:44:46.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:44:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:44:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:44:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:44:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:44:46.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:44:46.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:44:46.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:44:46.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:44:46.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:44:46.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:46.844+0000 7fc640db9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:44:46.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:46.846+0000 7fc640db9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:44:46.851 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:46.847+0000 7fc640db9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:44:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:44:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:44:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:47.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:44:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:47.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:47.708+0000 7fc640db9640 -1 Falling back to public interface 2024-09-10T20:44:48.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:48.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:48.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:44:48.383 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:44:48.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:48.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:44:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:44:49.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:44:49.338+0000 7fc640db9640 -1 osd.1 56 log_to_monitors true 2024-09-10T20:44:49.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:44:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:44:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:44:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:44:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:44:49.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 63 up_thru 36 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/2020220985,v1:127.0.0.1:6811/2020220985] [v2:127.0.0.1:6812/2020220985,v1:127.0.0.1:6813/2020220985] exists,up 2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:44:49.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:44:49.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:44:50.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:44:50.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:44:50.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:44:50.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:44:50.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:44:50.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:50.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:44:50.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-10T20:44:50.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-10T20:44:50.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-10T20:44:50.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:50.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:44:50.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939650 2024-09-10T20:44:50.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939650 2024-09-10T20:44:50.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-270582939650' 2024-09-10T20:44:50.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:50.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:44:50.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920780 2024-09-10T20:44:50.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920780 2024-09-10T20:44:50.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-270582939650 2-141733920780' 2024-09-10T20:44:50.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:50.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-10T20:44:50.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:50.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:44:50.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-10T20:44:50.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:50.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-10T20:44:50.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-10T20:44:50.873 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836504 2024-09-10T20:44:50.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:51.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-10T20:44:51.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:44:52.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:44:52.156 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:44:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-10T20:44:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:52.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939650 2024-09-10T20:44:52.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:52.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:44:52.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939650 2024-09-10T20:44:52.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:52.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939650 2024-09-10T20:44:52.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-09-10T20:44:52.521 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 270582939650 2024-09-10T20:44:52.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:44:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939650 -lt 270582939650 2024-09-10T20:44:52.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:52.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920780 2024-09-10T20:44:52.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:44:52.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920780 2024-09-10T20:44:52.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:52.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920780 2024-09-10T20:44:52.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920780' 2024-09-10T20:44:52.799 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 141733920780 2024-09-10T20:44:52.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:44:53.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920780 -lt 141733920780 2024-09-10T20:44:53.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:44:53.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:53.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:53.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:53.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T20:44:53.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:53.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:53.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:54.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-10T20:44:54.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T20:44:54.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:44:54.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T20:44:54.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:44:54.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:44:54.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:54.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:54.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:44:54.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:54.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:54.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:54.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:44:54.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 2 2024-09-10T20:44:54.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:44:54.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:44:54.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:44:54.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:54.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:54.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:55.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:44:55.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:55.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:55.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:55.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:44:55.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:44:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:44:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:44:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:44:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:44:55.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:44:56.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:44:56.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:44:56.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:44:56.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:44:56.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:44:56.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:44:56.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:44:56.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:44:56.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:44:56.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:44:56.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-10T20:44:56.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:44:56.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:44:56.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:44:56.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:44:57.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-10T20:44:57.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-10T20:44:57.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:44:57.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:44:57.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:44:57.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-10T20:44:37.162359+0000 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:44:37.162359+0000 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:44:57.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:44:57.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:44:57.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:44:37.162359+0000 '>' 2024-09-10T20:44:37.162359+0000 2024-09-10T20:44:57.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:44:58.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:44:58.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:44:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:44:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:44:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:44:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:44:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:44:58.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:44:57.508624+0000 '>' 2024-09-10T20:44:37.162359+0000 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:44:58.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:44:59.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:44:59.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:44:59.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:44:59.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:44:59.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:44:59.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:44:59.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:44:59.333 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:44:59.333 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:44:59.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:44:59.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:59.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:44:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-10T20:44:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-10T20:44:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-10T20:44:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:59.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:44:59.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939652 2024-09-10T20:44:59.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939652 2024-09-10T20:44:59.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-270582939652' 2024-09-10T20:44:59.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:44:59.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920783 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920783 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-270582939652 2-141733920783' 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-10T20:44:59.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:44:59.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:44:59.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-10T20:44:59.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:44:59.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-10T20:44:59.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-10T20:44:59.660 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836507 2024-09-10T20:44:59.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:45:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-10T20:45:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:45:00.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939652 2024-09-10T20:45:00.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:45:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:45:00.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939652 2024-09-10T20:45:00.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:45:00.021 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 270582939652 2024-09-10T20:45:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939652 2024-09-10T20:45:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939652' 2024-09-10T20:45:00.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:45:00.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939653 -lt 270582939652 2024-09-10T20:45:00.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:45:00.312 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920783 2024-09-10T20:45:00.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:45:00.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:45:00.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920783 2024-09-10T20:45:00.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:45:00.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920783 2024-09-10T20:45:00.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920783' 2024-09-10T20:45:00.315 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 141733920783 2024-09-10T20:45:00.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:45:00.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920783 -lt 141733920783 2024-09-10T20:45:00.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:45:00.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:00.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:45:01.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:45:01.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:45:01.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:45:01.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:45:01.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:45:01.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:01.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-10T20:45:01.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T20:45:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-10T20:45:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:45:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:45:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:45:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:45:01.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-10T20:45:01.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:45:01.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-10T20:45:02.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:45:02.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:45:02.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:45:02.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:45:02.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:45:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T20:45:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-10T20:45:03.080 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:45:03.080 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:45:03.080 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:45:03.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:45:03.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:45:03.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-10T20:45:03.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:45:03.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:45:03.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:45:03.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:45:03.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:45:03.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:03.379+0000 7f11b020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:03.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:03.438+0000 7f11b020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:03.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:03.439+0000 7f11b020b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:03.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:04.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:04.789 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:04.782+0000 7f11b020b640 -1 Falling back to public interface 2024-09-10T20:45:05.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:05.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:05.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:45:05.140 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:45:05.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:05.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:05.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:06.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:06.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:06.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:45:06.420 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:45:06.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:06.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:06.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:07.044 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:07.041+0000 7f11b020b640 -1 osd.1 72 log_to_monitors true 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:07.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 73 last_clean_interval [63,72) [v2:127.0.0.1:6810/3283162284,v1:127.0.0.1:6811/3283162284] [v2:127.0.0.1:6812/3283162284,v1:127.0.0.1:6813/3283162284] exists,up 2ab9a24e-df6e-4029-91e9-023b658b7dad 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:45:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:45:08.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:45:08.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:45:08.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:45:08.622 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:45:08.623 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T20:45:08.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:45:08.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:45:08.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:45:08.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-10T20:45:08.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-10T20:45:08.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-10T20:45:08.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:45:08.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:45:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-10T20:45:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-10T20:45:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-339302416386' 2024-09-10T20:45:09.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:45:09.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:45:09.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920786 2024-09-10T20:45:09.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920786 2024-09-10T20:45:09.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-339302416386 2-141733920786' 2024-09-10T20:45:09.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:45:09.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-10T20:45:09.211 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:45:09.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:45:09.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-10T20:45:09.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:45:09.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-10T20:45:09.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-10T20:45:09.213 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836510 2024-09-10T20:45:09.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:45:09.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-10T20:45:09.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:45:10.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:45:10.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:45:10.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-10T20:45:10.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:45:10.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-339302416386 2024-09-10T20:45:10.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:45:10.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:45:10.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-339302416386 2024-09-10T20:45:10.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:45:10.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-10T20:45:10.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-09-10T20:45:10.814 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 339302416386 2024-09-10T20:45:10.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:45:11.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416386 -lt 339302416386 2024-09-10T20:45:11.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:45:11.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920786 2024-09-10T20:45:11.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:45:11.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:45:11.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920786 2024-09-10T20:45:11.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:45:11.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920786 2024-09-10T20:45:11.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920786' 2024-09-10T20:45:11.098 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 141733920786 2024-09-10T20:45:11.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:45:11.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920786 -lt 141733920786 2024-09-10T20:45:11.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:45:11.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:11.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:45:11.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:45:12.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:45:12.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:45:12.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:12.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:12.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:45:12.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T20:45:12.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:45:12.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:45:12.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:45:12.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:45:12.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:45:12.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:45:12.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:45:12.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:45:12.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:45:12.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:45:12.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:45:12.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:45:12.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:45:12.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:45:12.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:12.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:45:13.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:45:13.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:45:13.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:45:13.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:45:13.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:45:13.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:45:13.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:45:13.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:45:13.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:45:13.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:45:13.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:45:13.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:45:13.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:45:13.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:45:13.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:45:13.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:45:13.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:45:13.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:45:14.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:45:14.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:45:14.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:45:14.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:45:14.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:45:14.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:45:14.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:45:14.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:45:14.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:45:14.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:45:14.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:45:14.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:45:14.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:45:14.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:45:14.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:45:14.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:45:14.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:45:14.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:45:14.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:45:14.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:45:14.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:45:14.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:14.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:14.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:45:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:45:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:45:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:45:14.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:45:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:45:14.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:45:14.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:45:14.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:45:14.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:45:14.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:45:14.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:45:14.438 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:45:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:45:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:45:14.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:45:14.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:45:14.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:45:14.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:45:14.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:45:14.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:45:14.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:14.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:14.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:45:14.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:45:14.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:45:14.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:45:14.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:45:14.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:14.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:14.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:45:14.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:45:14.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:45:14.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:45:14.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:45:14.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:45:14.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:45:14.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:14.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:45:15.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:45:15.074 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:45:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:45:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:45:15.075 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:45:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:45:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:45:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:45:15.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:45:15.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:45:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:16.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:45:16.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:45:16.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-10T20:45:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:45:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-10T20:45:16.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:45:16.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:45:16.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:45:16.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:45:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f3b2fe9b-1bed-444c-81d4-036c6e14e337 2024-09-10T20:45:16.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f3b2fe9b-1bed-444c-81d4-036c6e14e337' 2024-09-10T20:45:16.078 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 f3b2fe9b-1bed-444c-81d4-036c6e14e337 2024-09-10T20:45:16.078 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:45:16.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDcr+BmmQZcBRAA4zbM00hZ6QNhnWRPOwN7sA== 2024-09-10T20:45:16.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDcr+BmmQZcBRAA4zbM00hZ6QNhnWRPOwN7sA=="}' 2024-09-10T20:45:16.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f3b2fe9b-1bed-444c-81d4-036c6e14e337 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:45:16.298 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:45:16.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:45:16.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcr+BmmQZcBRAA4zbM00hZ6QNhnWRPOwN7sA== --osd-uuid f3b2fe9b-1bed-444c-81d4-036c6e14e337 2024-09-10T20:45:16.338 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:16.335+0000 7f8cf3ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:16.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:16.345+0000 7f8cf3ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:16.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:16.346+0000 7f8cf3ed6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:16.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:16.347+0000 7f8cf3ed6640 -1 bdev(0x55a8a4b69c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:45:16.350 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:16.347+0000 7f8cf3ed6640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:45:21.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:45:21.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:45:21.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:45:21.838 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:45:21.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:45:22.197 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:45:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:45:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:45:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:45:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:45:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:45:22.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:22.222+0000 7f4f4eda4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:22.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:22.235+0000 7f4f4eda4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:22.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:22.236+0000 7f4f4eda4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:22.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:45:22.473 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:45:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:22.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:22.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:23.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:23.597+0000 7f4f4eda4640 -1 Falling back to public interface 2024-09-10T20:45:23.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:23.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:23.750 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:45:23.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:45:23.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:23.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:24.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:25.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:25.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:25.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:45:25.042 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:45:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:25.170 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:25.167+0000 7f4f4eda4640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:45:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:26.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:26.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:26.339 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:45:26.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:45:26.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:26.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:26.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:27.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:45:27.626 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:45:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:27.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:27.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:28.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:28.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:28.914 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:45:28.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:45:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:29.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:30.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:30.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:30.202 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:45:30.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:45:30.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:30.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:30.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:31.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:31.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:31.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:45:31.491 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:45:31.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:31.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:31.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:32.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:32.927 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:45:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:45:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:33.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:34.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:34.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:34.216 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:45:34.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:45:34.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:34.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:34.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:35.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:35.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:35.505 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:45:35.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:45:35.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:35.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:36.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:37.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:37.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:37.093 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:45:37.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:45:37.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:37.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:37.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:38.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:38.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:38.544 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:45:38.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:45:38.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:38.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:39.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:40.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:40.327 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:45:40.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:40.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:45:40.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:40.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:40.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:41.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:41.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:41.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:45:41.908 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:45:41.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:41.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:42.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:43.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:43.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:43.627 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:45:43.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:45:43.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:43.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:44.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:45.401 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:45:45.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:45:45.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:45.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:46.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:47.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:47.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:47.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:45:47.095 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:45:47.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:47.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:45:47.379 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1396202640,v1:127.0.0.1:6803/1396202640] [v2:127.0.0.1:6804/1396202640,v1:127.0.0.1:6805/1396202640] exists,up f3b2fe9b-1bed-444c-81d4-036c6e14e337 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:45:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:45:47.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:45:47.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a909f2fa-8cca-463a-8ac4-b3a884be0d01' 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:45:47.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:45:47.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7r+BmGOzBFxAATEZ6OId3hW0uFRtFKbVXvw== 2024-09-10T20:45:47.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7r+BmGOzBFxAATEZ6OId3hW0uFRtFKbVXvw=="}' 2024-09-10T20:45:47.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a909f2fa-8cca-463a-8ac4-b3a884be0d01 -i td/osd-scrub-repair/1/new.json 2024-09-10T20:45:47.789 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:45:47.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:45:47.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7r+BmGOzBFxAATEZ6OId3hW0uFRtFKbVXvw== --osd-uuid a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:45:47.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:47.828+0000 7ff081dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:47.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:47.870+0000 7ff081dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:47.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:47.871+0000 7ff081dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:47.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:47.871+0000 7ff081dc5640 -1 bdev(0x5559c1cd3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:45:47.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:47.871+0000 7ff081dc5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:45:53.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:45:53.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:45:53.794 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:45:53.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:45:53.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:45:54.155 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:45:54.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:45:54.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:45:54.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:45:54.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:45:54.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:45:54.183 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:54.180+0000 7f5f48f54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:54.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:54.200+0000 7f5f48f54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:54.205 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:54.201+0000 7f5f48f54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:45:54.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:45:54.432 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:45:54.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:54.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:54.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:55.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:55.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:55.710 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:45:55.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:45:55.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:55.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:55.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:56.047 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:56.044+0000 7f5f48f54640 -1 Falling back to public interface 2024-09-10T20:45:56.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:56.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:56.999 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:45:57.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:45:57.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:57.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:57.879 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:45:57.875+0000 7f5f48f54640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:45:58.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:58.294 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:45:58.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:58.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:45:58.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:58.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:45:59.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:45:59.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:45:59.633 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:45:59.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:45:59.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:45:59.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:45:59.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:00.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:00.921 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:46:00.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:00.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:46:00.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:00.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:01.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:02.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:02.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:02.294 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:46:02.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:46:02.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:02.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:02.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:03.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:03.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:03.650 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:46:03.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:46:03.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:03.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:04.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:04.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:04.939 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:46:04.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:46:04.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:04.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:05.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:06.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:06.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:06.229 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:46:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:46:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:06.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:06.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:07.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:07.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:07.592 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:46:07.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:46:07.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:07.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:07.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:08.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:08.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:08.879 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:46:08.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:46:08.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:08.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:09.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:10.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:10.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:10.167 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:46:10.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:46:10.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:10.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:11.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:11.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:11.556 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:46:11.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:46:11.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:11.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:13.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:13.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:13.058 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:46:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:46:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:13.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:13.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:14.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:14.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:14.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:46:14.629 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:46:14.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:14.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:15.918 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:46:15.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:46:15.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:15.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:16.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:17.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:17.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:17.306 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:46:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:46:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:17.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:18.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:18.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:18.659 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:46:18.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:46:18.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:18.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:18.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:19.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:19.993 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:46:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:46:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:19.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:20.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:21.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:21.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:21.323 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T20:46:21.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T20:46:21.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:21.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:46:21.603 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2984389256,v1:127.0.0.1:6811/2984389256] [v2:127.0.0.1:6812/2984389256,v1:127.0.0.1:6813/2984389256] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:46:21.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:46:21.609 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:46:21.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:46:21.611 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:46:21.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f7874cec-78da-406d-b281-39449a6708b1' 2024-09-10T20:46:21.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:46:21.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdsOBmpiApJRAAjtm2W2llCAcGTLL/Ad9etw== 2024-09-10T20:46:21.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdsOBmpiApJRAAjtm2W2llCAcGTLL/Ad9etw=="}' 2024-09-10T20:46:21.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f7874cec-78da-406d-b281-39449a6708b1 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:46:21.987 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:46:21.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:46:22.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdsOBmpiApJRAAjtm2W2llCAcGTLL/Ad9etw== --osd-uuid f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:46:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:22.025+0000 7fed6d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:22.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:22.075+0000 7fed6d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:22.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:22.076+0000 7fed6d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:22.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:22.077+0000 7fed6d20b640 -1 bdev(0x5611e130dc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:46:22.081 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:22.077+0000 7fed6d20b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:46:27.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:46:27.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:46:27.978 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:46:27.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:46:27.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:46:28.340 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:46:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:46:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:46:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:46:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:46:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:46:28.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:28.363+0000 7fc22e20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:28.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:28.366+0000 7fc22e20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:28.371 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:28.367+0000 7fc22e20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:46:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:28.617 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:46:28.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:46:28.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:28.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:28.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:29.742 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:29.737+0000 7fc22e20d640 -1 Falling back to public interface 2024-09-10T20:46:29.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:29.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:29.897 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:46:29.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:46:29.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:29.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:30.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:31.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:31.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:31.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:46:31.188 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:46:31.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:31.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:31.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:31.425+0000 7fc22e20d640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:46:31.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:32.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:32.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:32.477 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:46:32.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:46:32.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:32.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:33.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:33.763 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:46:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:46:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:33.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:34.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:35.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:35.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:35.048 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:46:35.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:46:35.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:35.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:35.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:36.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:36.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:36.412 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:46:36.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:46:36.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:36.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:36.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:37.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:37.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:37.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:46:37.793 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:46:37.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:37.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:38.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:39.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:39.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:39.082 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:46:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:46:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:39.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:40.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:40.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:40.490 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:46:40.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:46:40.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:40.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:40.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:41.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:41.817 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:46:41.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:41.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:46:41.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:41.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:42.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:43.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:43.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:43.268 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:46:43.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:46:43.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:43.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:44.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:44.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:44.727 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:46:44.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:46:44.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:44.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:45.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:46.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:46.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:46.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:46:46.057 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:46:46.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:46.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:46.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:47.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:47.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:47.509 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:46:47.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:46:47.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:47.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:48.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:49.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:49.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:49.187 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:46:49.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:46:49.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:49.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:49.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:50.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:50.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:50.965 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:46:50.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:46:50.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:50.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:51.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:52.654 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:46:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:46:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:53.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:46:54.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:46:54.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:46:54.337 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:46:54.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:46:54.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:46:54.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:46:54.621 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1056644895,v1:127.0.0.1:6819/1056644895] [v2:127.0.0.1:6820/1056644895,v1:127.0.0.1:6821/1056644895] exists,up f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:46:54.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:46:54.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:46:54.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:46:54.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:46:54.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:46:54.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:46:54.627 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:46:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1d4cfff0-7220-4b2c-8a04-0a84a04899a9' 2024-09-10T20:46:54.629 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:46:54.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:46:54.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+sOBmciIoJhAArWJ3A8lW0INI2NDW+PlKEQ== 2024-09-10T20:46:54.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+sOBmciIoJhAArWJ3A8lW0INI2NDW+PlKEQ=="}' 2024-09-10T20:46:54.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 -i td/osd-scrub-repair/3/new.json 2024-09-10T20:46:55.062 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:46:55.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T20:46:55.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+sOBmciIoJhAArWJ3A8lW0INI2NDW+PlKEQ== --osd-uuid 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:46:55.105 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:55.101+0000 7f01f01cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:55.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:55.147+0000 7f01f01cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:55.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:55.148+0000 7f01f01cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:46:55.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:55.149+0000 7f01f01cc640 -1 bdev(0x55c7d9f77c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:46:55.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:46:55.149+0000 7f01f01cc640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T20:47:01.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T20:47:01.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:47:01.196 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T20:47:01.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:47:01.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:47:01.555 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T20:47:01.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:47:01.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:47:01.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:47:01.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:47:01.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:47:01.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:01.579+0000 7fe467219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:01.584 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:01.581+0000 7fe467219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:01.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:01.582+0000 7fe467219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:01.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:47:01.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:47:01.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:01.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:02.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:03.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:03.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:03.125 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:47:03.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:47:03.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:03.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:03.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:03.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:03.447+0000 7fe467219640 -1 Falling back to public interface 2024-09-10T20:47:04.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:04.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:04.411 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:47:04.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:47:04.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:04.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:04.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:05.538 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:05.534+0000 7fe467219640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:47:05.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:05.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:05.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:47:05.690 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:47:05.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:05.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:06.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:07.024 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:47:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:47:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:07.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:08.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:08.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:08.382 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:47:08.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:47:08.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:08.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:08.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:09.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:09.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:09.805 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:47:09.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:47:09.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:09.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:10.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:11.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:11.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:11.147 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:47:11.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:47:11.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:11.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:12.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:12.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:12.537 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:47:12.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:47:12.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:12.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:12.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:13.911 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:47:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:47:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:14.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:15.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:15.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:15.409 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:47:15.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:47:15.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:15.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:15.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:16.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:16.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:16.702 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:47:16.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:47:16.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:16.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:16.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:17.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:17.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:17.993 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:47:17.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:47:17.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:17.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:18.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:19.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:19.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:19.792 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:47:19.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:47:19.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:19.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:20.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:21.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:21.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:21.469 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:47:21.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:47:21.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:21.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:22.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:23.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:23.172 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:47:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:47:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:23.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:23.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:24.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:24.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:24.682 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:47:24.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:47:24.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:24.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:25.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:26.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:26.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:26.393 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:47:26.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:47:26.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:26.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:27.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:28.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:28.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:28.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:47:28.188 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:47:28.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:28.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:28.601 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/465299661,v1:127.0.0.1:6827/465299661] [v2:127.0.0.1:6828/465299661,v1:127.0.0.1:6829/465299661] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:47:28.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:47:28.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:47:28.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:47:28.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-10T20:47:28.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T20:47:28.883 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T20:47:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T20:47:28.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T20:47:29.265 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T20:47:29.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:47:30.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T20:47:32.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-10T20:47:32.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:47:32.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:47:32.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:47:32.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:47:32.809 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:47:32.809 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:47:32.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:47:32.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:47:32.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:47:32.908 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:47:33.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:33.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:47:33.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-10T20:47:33.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-10T20:47:33.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-10T20:47:33.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:33.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:47:33.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T20:47:33.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T20:47:33.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-09-10T20:47:33.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:33.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:47:33.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-10T20:47:33.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-10T20:47:33.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509460' 2024-09-10T20:47:33.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:33.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:47:33.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-10T20:47:33.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-10T20:47:33.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509460 3-85899345926' 2024-09-10T20:47:33.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:33.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-10T20:47:33.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:33.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:47:33.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-10T20:47:33.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:33.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-10T20:47:33.593 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836508 2024-09-10T20:47:33.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-10T20:47:33.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:47:33.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-10T20:47:33.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:47:34.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:47:34.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:47:35.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-10T20:47:35.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:35.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T20:47:35.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:35.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:47:35.156 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T20:47:35.156 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:35.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T20:47:35.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T20:47:35.157 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672982 2024-09-10T20:47:35.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:47:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-10T20:47:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:35.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-10T20:47:35.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:47:35.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-10T20:47:35.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:35.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-10T20:47:35.437 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509460 2024-09-10T20:47:35.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-10T20:47:35.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:47:35.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509460 2024-09-10T20:47:35.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:35.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-10T20:47:35.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:35.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:47:35.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-10T20:47:35.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:35.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-10T20:47:35.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-10T20:47:35.715 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345926 2024-09-10T20:47:35.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:47:35.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345927 -lt 85899345926 2024-09-10T20:47:35.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:47:35.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:35.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:47:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:47:36.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:47:36.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:47:36.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:36.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:47:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:47:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T20:47:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-10T20:47:37.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T20:47:37.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T20:47:38.079 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T20:47:38.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:47:39.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T20:47:39.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T20:47:39.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:47:39.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:47:39.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:47:39.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:47:39.094 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:47:39.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:47:39.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:47:39.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:47:39.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:47:39.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:39.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:47:39.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-10T20:47:39.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-10T20:47:39.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-10T20:47:39.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:39.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:47:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-10T20:47:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-10T20:47:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984' 2024-09-10T20:47:39.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:39.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:47:39.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-10T20:47:39.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-10T20:47:39.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509463' 2024-09-10T20:47:39.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:39.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345929 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345929 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672984 2-64424509463 3-85899345929' 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-10T20:47:39.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:40.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:47:40.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-10T20:47:40.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:40.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-10T20:47:40.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-10T20:47:40.001 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836510 2024-09-10T20:47:40.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:47:40.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-10T20:47:40.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:40.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-10T20:47:40.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:40.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:47:40.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-10T20:47:40.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:40.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-10T20:47:40.281 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672984 2024-09-10T20:47:40.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-10T20:47:40.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:47:40.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-10T20:47:40.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:40.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-10T20:47:40.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:40.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:47:40.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-10T20:47:40.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:40.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-10T20:47:40.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-10T20:47:40.576 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509463 2024-09-10T20:47:40.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:47:40.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-10T20:47:40.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:40.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345929 2024-09-10T20:47:40.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:40.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:47:40.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345929 2024-09-10T20:47:40.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:40.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345929 2024-09-10T20:47:40.861 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345929 2024-09-10T20:47:40.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-09-10T20:47:40.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:47:41.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345929 -lt 85899345929 2024-09-10T20:47:41.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:47:41.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:41.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:47:41.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:47:41.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:47:41.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:47:41.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:41.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:42.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T20:47:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T20:47:42.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T20:47:42.478 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T20:47:42.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T20:47:43.014 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T20:47:43.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T20:47:43.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T20:47:43.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:47:43.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T20:47:43.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:47:43.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:47:43.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:47:43.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:47:43.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-10T20:47:43.648 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T20:47:43.648 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T20:47:43.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T20:47:43.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-10T20:47:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:47:43.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-10T20:47:43.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:47:43.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:47:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:47:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T20:47:46.098 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T20:47:46.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:47:46.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:47:46.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:47:46.626 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:47:46.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:47:46.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:46.651+0000 7f25d4e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:46.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:46.654+0000 7f25d4e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:46.658 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:46.655+0000 7f25d4e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:47:46.913 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:47:46.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:46.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:47.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:47.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:47.754+0000 7f25d4e33640 -1 Falling back to public interface 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:48.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:48.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:49.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:49.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:49.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:47:49.476 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:47:49.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:49.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:49.637 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:47:49.633+0000 7f25d4e33640 -1 osd.3 36 log_to_monitors true 2024-09-10T20:47:49.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:47:50.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/2694714956,v1:127.0.0.1:6827/2694714956] [v2:127.0.0.1:6828/2694714956,v1:127.0.0.1:6829/2694714956] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:47:51.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:47:51.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:47:51.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:47:51.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:47:51.628 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:47:51.628 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:47:51.629 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:47:51.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:47:51.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:51.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:47:51.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-10T20:47:51.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-10T20:47:51.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-10T20:47:51.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:51.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:47:51.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-10T20:47:51.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-10T20:47:51.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988' 2024-09-10T20:47:51.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:51.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:47:51.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-10T20:47:51.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-10T20:47:51.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988 2-64424509466' 2024-09-10T20:47:51.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:47:51.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:47:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-10T20:47:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-10T20:47:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672988 2-64424509466 3-176093659138' 2024-09-10T20:47:52.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-10T20:47:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:52.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:47:52.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-10T20:47:52.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-10T20:47:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-10T20:47:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836514 2024-09-10T20:47:52.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:47:52.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-10T20:47:52.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:52.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-10T20:47:52.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:52.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:47:52.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-10T20:47:52.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:52.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-10T20:47:52.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-10T20:47:52.372 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672988 2024-09-10T20:47:52.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:47:52.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-10T20:47:52.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:52.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-10T20:47:52.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:52.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:47:52.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-10T20:47:52.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:52.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-10T20:47:52.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-10T20:47:52.690 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509466 2024-09-10T20:47:52.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:47:52.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-10T20:47:52.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:47:52.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659138 2024-09-10T20:47:52.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:47:52.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:47:52.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659138 2024-09-10T20:47:52.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:47:52.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-10T20:47:52.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-09-10T20:47:52.969 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 176093659138 2024-09-10T20:47:52.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:47:53.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 176093659138 2024-09-10T20:47:53.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:47:54.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:47:54.248 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:47:54.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-10T20:47:54.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:47:54.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:54.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:47:54.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:47:55.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:47:55.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:47:55.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:47:55.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:47:55.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:47:55.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:47:55.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:47:55.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T20:47:55.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:47:55.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:47:55.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:47:55.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:47:55.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T20:47:55.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:47:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:47:56.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:56.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:47:56.327 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:47:56.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:47:56.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:47:56.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:47:56.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:47:56.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:47:56.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:37.950487+0000 '>' 2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:56.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:47:57.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:47:57.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:47:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:37.950487+0000 '>' 2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:57.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:47:58.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:47:58.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:47:58.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:47:58.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:47:58.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:47:58.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:47:58.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:47:59.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:37.950487+0000 '>' 2024-09-10T20:47:37.950487+0000 2024-09-10T20:47:59.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:00.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:00.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:00.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:00.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:00.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:00.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:00.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:56.638030+0000 '>' 2024-09-10T20:47:37.950487+0000 2024-09-10T20:48:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:48:00.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:48:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:48:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:48:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:48:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:48:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:48:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:48:03.360 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:48:03.360 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:48:03.361 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:48:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:48:03.630 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:48:03.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:48:03.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:48:03.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:48:03.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:48:03.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:48:03.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:48:03.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:03.655+0000 7f704e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:03.674 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:03.670+0000 7f704e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:03.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:03.671+0000 7f704e617640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:03.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:48:03.923 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:48:03.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:03.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:04.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:04.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:04.794+0000 7f704e617640 -1 Falling back to public interface 2024-09-10T20:48:05.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:05.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:05.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:48:05.199 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:05.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:05.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:05.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:06.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:06.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:06.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:48:06.488 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:06.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:06.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:06.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:07.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:06.999+0000 7f704e617640 -1 osd.3 43 log_to_monitors true 2024-09-10T20:48:07.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:48:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:48:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:08.197 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 47 up_thru 47 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/44603466,v1:127.0.0.1:6827/44603466] [v2:127.0.0.1:6828/44603466,v1:127.0.0.1:6829/44603466] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:48:08.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:48:08.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:48:08.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:48:08.605 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:08.605 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:08.605 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:48:08.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:48:08.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:08.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:48:08.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836518 2024-09-10T20:48:08.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836518 2024-09-10T20:48:08.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518' 2024-09-10T20:48:08.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:08.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:48:08.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672992 2024-09-10T20:48:08.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672992 2024-09-10T20:48:08.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672992' 2024-09-10T20:48:08.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:08.911 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:48:09.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-10T20:48:09.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-10T20:48:09.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672992 2-64424509471' 2024-09-10T20:48:09.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:09.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:48:09.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462914 2024-09-10T20:48:09.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462914 2024-09-10T20:48:09.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672992 2-64424509471 3-201863462914' 2024-09-10T20:48:09.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:09.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836518 2024-09-10T20:48:09.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:09.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:48:09.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836518 2024-09-10T20:48:09.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:09.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836518 2024-09-10T20:48:09.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-10T20:48:09.120 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836518 2024-09-10T20:48:09.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:09.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836518 2024-09-10T20:48:09.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:48:10.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:48:10.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:10.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-10T20:48:10.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:10.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672992 2024-09-10T20:48:10.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:10.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:48:10.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672992 2024-09-10T20:48:10.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:10.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672992 2024-09-10T20:48:10.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-10T20:48:10.687 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672992 2024-09-10T20:48:10.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:48:10.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-10T20:48:10.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:10.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-10T20:48:10.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:10.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:48:10.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-10T20:48:10.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:10.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-10T20:48:10.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-10T20:48:10.967 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509471 2024-09-10T20:48:10.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:48:11.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-10T20:48:11.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:11.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-201863462914 2024-09-10T20:48:11.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:11.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:48:11.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-201863462914 2024-09-10T20:48:11.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:11.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462914 2024-09-10T20:48:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-09-10T20:48:11.247 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 201863462914 2024-09-10T20:48:11.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:48:11.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462914 -lt 201863462914 2024-09-10T20:48:11.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:48:11.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:11.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:11.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:48:11.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:11.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:12.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:48:12.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:12.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:12.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:12.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:48:12.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:48:12.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:48:12.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:48:12.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:48:12.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:48:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:48:12.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:48:14.890 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:48:15.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:48:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:48:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:48:15.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:48:15.414 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:48:15.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:48:15.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:48:15.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:48:15.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:48:15.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:48:15.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:48:15.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:15.438+0000 7f82071dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:15.445 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:15.441+0000 7f82071dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:15.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:15.443+0000 7f82071dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:15.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:48:15.702 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:48:15.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:15.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:15.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:16.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:16.796+0000 7f82071dd640 -1 Falling back to public interface 2024-09-10T20:48:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:48:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:17.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:18.192 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:18.188+0000 7f82071dd640 -1 osd.1 49 log_to_monitors true 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:18.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:18.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:19.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:48:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:48:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:19.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:19.900 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/339196576,v1:127.0.0.1:6811/339196576] [v2:127.0.0.1:6812/339196576,v1:127.0.0.1:6813/339196576] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:48:19.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:48:19.902 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:48:19.902 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:48:19.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:48:19.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:48:19.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:48:19.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:48:19.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:48:19.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:48:19.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:48:19.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:48:19.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:20.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:48:20.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-10T20:48:20.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-10T20:48:20.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-10T20:48:20.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:20.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:48:20.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233986 2024-09-10T20:48:20.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233986 2024-09-10T20:48:20.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-231928233986' 2024-09-10T20:48:20.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:20.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:48:20.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509474 2024-09-10T20:48:20.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509474 2024-09-10T20:48:20.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-231928233986 2-64424509474' 2024-09-10T20:48:20.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:20.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:48:20.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462917 2024-09-10T20:48:20.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462917 2024-09-10T20:48:20.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-231928233986 2-64424509474 3-201863462917' 2024-09-10T20:48:20.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:20.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-10T20:48:20.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:20.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:48:20.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-10T20:48:20.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:20.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-10T20:48:20.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-10T20:48:20.801 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836521 2024-09-10T20:48:20.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:21.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836521 2024-09-10T20:48:21.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:48:22.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:48:22.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:22.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836521 2024-09-10T20:48:22.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:22.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-231928233986 2024-09-10T20:48:22.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:22.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:48:22.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-231928233986 2024-09-10T20:48:22.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:22.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233986 2024-09-10T20:48:22.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-09-10T20:48:22.382 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 231928233986 2024-09-10T20:48:22.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:48:22.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233986 -lt 231928233986 2024-09-10T20:48:22.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:22.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509474 2024-09-10T20:48:22.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:22.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:48:22.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509474 2024-09-10T20:48:22.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:22.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509474 2024-09-10T20:48:22.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-10T20:48:22.662 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509474 2024-09-10T20:48:22.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:48:22.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-10T20:48:22.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:22.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-201863462917 2024-09-10T20:48:22.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:22.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:48:22.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-201863462917 2024-09-10T20:48:22.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:22.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462917 2024-09-10T20:48:22.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 201863462917' 2024-09-10T20:48:22.942 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 201863462917 2024-09-10T20:48:22.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:48:23.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462917 -lt 201863462917 2024-09-10T20:48:23.219 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:48:23.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:23.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:23.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:23.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:23.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:48:23.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:23.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:23.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:24.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:48:24.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:48:24.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:24.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:24.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:24.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:48:25.021 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:48:25.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:25.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:48:25.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:25.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:25.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:25.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:56.638030+0000 '>' 2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:25.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:26.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:26.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:26.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:26.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:26.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:26.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:26.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:26.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:56.638030+0000 '>' 2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:26.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:27.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:27.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:27.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:27.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:27.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:27.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:27.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:47:56.638030+0000 '>' 2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:27.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:28.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:28.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:28.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:28.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:28.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:29.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:47:56.638030+0000 2024-09-10T20:48:29.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:48:29.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:48:31.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:48:31.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:48:31.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:48:31.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:48:31.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:48:31.930 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:48:31.931 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:48:31.931 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:48:31.931 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:48:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:32.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:48:32.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:48:32.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:48:32.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:48:32.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:32.233+0000 7fc545ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:32.239 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:32.235+0000 7fc545ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:32.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:32.237+0000 7fc545ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:32.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:32.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:32.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:33.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:33.093+0000 7fc545ddb640 -1 Falling back to public interface 2024-09-10T20:48:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:48:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:34.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:35.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:48:35.087+0000 7fc545ddb640 -1 osd.3 56 log_to_monitors true 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:35.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:35.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:36.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:48:36.977 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/3623978676,v1:127.0.0.1:6827/3623978676] [v2:127.0.0.1:6828/3623978676,v1:127.0.0.1:6829/3623978676] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:48:36.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:48:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:48:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:48:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:48:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:48:37.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:48:37.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:37.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:48:37.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836526 2024-09-10T20:48:37.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836526 2024-09-10T20:48:37.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526' 2024-09-10T20:48:37.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:37.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:48:37.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233990 2024-09-10T20:48:37.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233990 2024-09-10T20:48:37.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-231928233990' 2024-09-10T20:48:37.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:37.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:48:37.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509478 2024-09-10T20:48:37.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509478 2024-09-10T20:48:37.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-231928233990 2-64424509478' 2024-09-10T20:48:37.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:37.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037762 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037762 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-231928233990 2-64424509478 3-257698037762' 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836526 2024-09-10T20:48:37.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:37.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:48:37.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836526 2024-09-10T20:48:37.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:37.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836526 2024-09-10T20:48:37.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-10T20:48:37.757 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836526 2024-09-10T20:48:37.757 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:38.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836526 2024-09-10T20:48:38.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:48:39.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:48:39.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:39.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-10T20:48:39.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:39.312 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-231928233990 2024-09-10T20:48:39.312 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:39.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:48:39.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-231928233990 2024-09-10T20:48:39.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233990 2024-09-10T20:48:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 231928233990' 2024-09-10T20:48:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 231928233990 2024-09-10T20:48:39.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:48:39.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233990 -lt 231928233990 2024-09-10T20:48:39.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:39.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509478 2024-09-10T20:48:39.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:48:39.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509478 2024-09-10T20:48:39.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:39.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509478 2024-09-10T20:48:39.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-10T20:48:39.594 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509478 2024-09-10T20:48:39.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:48:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-10T20:48:39.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:39.869 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037762 2024-09-10T20:48:39.869 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:39.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:48:39.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037762 2024-09-10T20:48:39.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037762 2024-09-10T20:48:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-09-10T20:48:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 257698037762 2024-09-10T20:48:39.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:48:40.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037762 -lt 257698037762 2024-09-10T20:48:40.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:48:40.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:40.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:48:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:40.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:40.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:48:40.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:40.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:40.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:41.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:48:41.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:48:41.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:48:41.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:48:41.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:48:41.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 147033"' 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 147033' 2024-09-10T20:48:41.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 147035"' 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 147035' 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 147033 147035' 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 147033 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/147036: /' 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:48:41.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/147038: /' 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:48:44.233 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:48:44.234 INFO:tasks.workunit.client.0.smithi181.stderr:147036: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:48:44.235 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:48:44.236 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:48:44.236 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:48:44.236 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:48:44.236 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:48:44.236 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:48:44.237 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: start osd.1 2024-09-10T20:48:44.238 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:48:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:1147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.327 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:48:44.328 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-10T20:48:44.329 INFO:tasks.workunit.client.0.smithi181.stderr:147038: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:48:44.330 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:48:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:48:44.332 INFO:tasks.workunit.client.0.smithi181.stderr:147038: start osd.2 2024-09-10T20:48:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:48:48.743 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:48:48.743 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:48:48.743 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2024-09-10T20:48:44.263+0000 7fbf0c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2024-09-10T20:48:44.265+0000 7fbf0c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2024-09-10T20:48:44.267+0000 7fbf0c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:48:48.744 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 0 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:48:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 1 2024-09-10T20:48:48.746 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2024-09-10T20:48:45.860+0000 7fbf0c41e640 -1 Falling back to public interface 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.748 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2024-09-10T20:48:48.125+0000 7fbf0c41e640 -1 osd.1 62 log_to_monitors true 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.749 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.750 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:48:48.750 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 3 2024-09-10T20:48:48.750 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.750 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147036: osd.1 up in weight 1 up_from 68 up_thru 57 down_at 63 last_c47038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2024-09-10T20:48:44.361+0000 7f15657dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2024-09-10T20:48:44.363+0000 7f15657dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2024-09-10T20:48:44.364+0000 7f15657dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:48:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 0 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2024-09-10T20:48:45.489+0000 7f15657dd640 -1 Falling back to public interface 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 1 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:48:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2024-09-10T20:48:47.843+0000 7f15657dd640 -1 osd.2 62 log_to_monitors true 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 3 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:48:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147038: osd.2 up in weight 1 up_from 68 up_thru 50 down_at 63 last_clean_interval [54,62) [v2:127.0.0.1:6818/1987566425,v1:127.0.0.1:6819/1987566425] [v2:127.0.0.1:6820/1987566425,v1:127.0.0.1:6821/1987566425] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:48:49.714 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:48:49.715 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 1 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 2 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: 3' 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:49.716 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836530 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836530 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530' 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-292057776130' 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tlean_interval [15,62) [v2:127.0.0.1:6810/3612405009,v1:127.0.0.1:6811/3612405009] [v2:127.0.0.1:6812/3612405009,v1:127.0.0.1:6813/3612405009] exists,up f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:48:49.717 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:48:49.718 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:48:49.719 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 1 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 2 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: 3' 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776131 2024-09-10T20:48:49.720 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776131 2024-09-10T20:48:49.721 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-292057776131' 2024-09-10T20:48:49.721 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:50.613 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-292057776131 2-292057776130' 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037765 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037765 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-292057776131 2-292057776130 3-257698037765' 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:48:50.614 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: waiting osd.0 seq 21474836529 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836529 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776131 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776131 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:50.615 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776131 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: waiting osd.1 seq 292057776131 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776131 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-292057776130 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:48:50.616 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-292057776130 2024-09-10T20:48:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-09-10T20:48:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776131 2024-09-10T20:48:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776131 2024-09-10T20:48:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-292057776130 2-292057776131' 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037766 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037766 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-292057776130 2-292057776131 3-257698037766' 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836530 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836530 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836530 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-10T20:48:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:147036: waiting osd.0 seq 21474836530 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776130 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776130 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: waiting osd.1 seq 292057776130 2024-09-10T20:48:50.620 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776130 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-292057776131 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:48:50.621 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-292057776131 2024-09-10T20:48:52.200 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: waiting osd.2 seq 292057776130 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776130 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037765 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037765 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:52.201 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037765 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037765' 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: waiting osd.3 seq 257698037765 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037766 -lt 257698037765 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:52.202 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-10T20:48:52.203 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standaloneest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776131 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: waiting osd.2 seq 292057776131 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776131 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037766 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:48:52.269 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037766 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037766 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: waiting osd.3 seq 257698037766 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037766 -lt 257698037766 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-10T20:48:52.271 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T20:48:53.376 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:48:53.376 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T20:48:53.376 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:53.377 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 2 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:53.378 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:53.379 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:53.379 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:48:53.379 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: /ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:53.383 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 2 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:48:53.384 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:48:53.385 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:54.955 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:48:54.956 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:54.957 INFO:tasks.workunit.client.0.smithi181.stderr:147038: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:54.958 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:48:54.958 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:48:54.958 INFO:tasks.workunit.client.0.smithi181.stderr:147038: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:48:54.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr: ceph --format json status 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:48:54.979 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:48:54.980 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:147036: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:48:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 147035 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:48:54.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:48:55.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:55.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:55.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:55.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:48:55.847 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:55.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:55.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:56.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:57.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:57.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:57.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:57.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:57.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:57.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:57.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:57.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:58.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:58.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:48:58.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:48:58.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:48:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:48:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:48:59.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:48:59.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:48:59.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:48:59.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:48:59.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:49:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:01.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:01.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:01.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:01.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:01.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:01.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:25.089356+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:49:01.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:02.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:02.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:02.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:02.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:02.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:02.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:02.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:02.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:25.089356+0000 2024-09-10T20:49:02.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:49:02.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 150211"' 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 150211' 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 150213"' 2024-09-10T20:49:04.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 150213' 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 150211 150213' 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 150211 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/150214: /' 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:49:04.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/150216: /' 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:05.779 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:05.780 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: _ 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: hinfo_key 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: snapset 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:05.781 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:05.782 INFO:tasks.workunit.client.0.smithi181.stderr:150214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:49:05.784 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:49:05.784 INFO:tasks.workunit.client.0.smithi181.stderr:150214: start osd.1 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:05.828 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:49:05.829 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: _ 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: hinfo_key 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: snapset 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:05.830 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:05.831 INFO:tasks.workunit.client.0.smithi181.stderr:150216: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:05.833 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:49:05.833 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:49:05.833 INFO:tasks.workunit.client.0.smithi181.stderr:150216: start osd.2 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2024-09-10T20:49:05.810+0000 7fc31e7a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2024-09-10T20:49:05.812+0000 7fc31e7a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2024-09-10T20:49:05.814+0000 7fc31e7a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 0 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2024-09-10T20:49:06.907+0000 7fc31e7a2640 -1 Falling back to public interface 2024-09-10T20:49:10.673 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 1 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.674 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2024-09-10T20:49:09.277+0000 7fc31e7a2640 -1 osd.1 69 log_to_monitors true 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 3 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150214: osd.1 up in weight 1 up_from 74 up_thru 74 down_atvery_ops 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T20:49:10.675 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2024-09-10T20:49:05.856+0000 7f60b663e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2024-09-10T20:49:05.930+0000 7f60b663e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2024-09-10T20:49:05.931+0000 7f60b663e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:10.676 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 0 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2024-09-10T20:49:07.267+0000 7f60b663e640 -1 Falling back to public interface 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 1 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:49:10.677 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2024-09-10T20:49:09.779+0000 7f60b663e640 -1 osd.2 69 log_to_monitors true 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:10.678 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:10.679 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:10.679 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 3 2024-09-10T20:49:10.679 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:10.679 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:49:11.557 INFO:tasks.workunit.client.0.smithi181.stderr:150216: osd.2 up in weight 1 up_from 76 up_thru 50 down_at 70 last_clean_interval [68,69) [v2:127.0.0.1:6810/4231572284,v1:127.0.0.1:6811/4231572284] [v2:127.0.0.1:6812/4231572284,v1:127.0.0.1:6813/4231572284] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:11.558 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 1 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 2 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: 3' 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:11.559 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836535 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836535 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535' 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-317827579906' 2024-09-10T20:49:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stat 70 last_clean_interval [68,69) [v2:127.0.0.1:6818/3381333550,v1:127.0.0.1:6819/3381333550] [v2:127.0.0.1:6820/3381333550,v1:127.0.0.1:6821/3381333550] exists,up f7874cec-78da-406d-b281-39449a6708b1 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 1 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 2 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: 3' 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579907 2024-09-10T20:49:11.766 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579907 2024-09-10T20:49:11.767 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-317827579907' 2024-09-10T20:49:11.767 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:12.583 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514498 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514498 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-317827579906 2-326417514498' 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:12.584 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037772 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037772 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-317827579906 2-326417514498 3-257698037772' 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836535 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.585 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836535 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836535 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: waiting osd.0 seq 21474836535 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:12.586 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836535 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-317827579906 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-317827579906 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:12.587 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: waiting osd.1 seq 317827579906 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579907 -lt 317827579906 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.588 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514498 2024-09-10T20:49:12.589 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.589 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:12.589 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514498 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubs: ceph tell osd.2 flush_pg_stats 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514499 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514499 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-317827579907 2-326417514499' 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:12.749 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037773 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037773 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-317827579907 2-326417514499 3-257698037773' 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: waiting osd.0 seq 21474836536 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836536 2024-09-10T20:49:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-317827579907 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-317827579907 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579907 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: waiting osd.1 seq 317827579907 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579907 -lt 317827579907 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514499 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:12.751 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:12.752 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514499 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514498 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: waiting osd.2 seq 326417514498 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514499 -lt 326417514498 2024-09-10T20:49:14.218 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037772 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037772 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037772 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: waiting osd.3 seq 257698037772 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037772 -lt 257698037772 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:14.219 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-10T20:49:14.220 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' -1 2024-09-10T20:49:15.348 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:15.348 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514499 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: waiting osd.2 seq 326417514499 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514499 -lt 326417514499 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037773 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037773 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037773 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-09-10T20:49:15.349 INFO:tasks.workunit.client.0.smithi181.stderr:150216: waiting osd.3 seq 257698037773 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037772 -lt 257698037773 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037773 -lt 257698037773 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:15.350 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:49:15.351 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtstandalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:15.396 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 1 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:49:15.397 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:49:15.398 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: getest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:150216: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:_num_pgs: ceph --format json status 2024-09-10T20:49:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:49:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T20:49:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:49:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:150214: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 150213 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:49:16.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:49:16.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:16.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 152725"' 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 152725' 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 152727"' 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 152727' 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 152725 152727' 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 152725 2024-09-10T20:49:16.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/152728: /' 2024-09-10T20:49:16.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:49:16.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/152730: /' 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.131 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:20.132 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:20.133 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:49:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:49:20.134 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:20.135 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: start osd.3 2024-09-10T20:49:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:49:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:1152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:20.351 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:20.352 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:49:20.353 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:20.354 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: start osd.1 2024-09-10T20:49:20.355 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:24.687 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2024-09-10T20:49:20.161+0000 7f0606e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2024-09-10T20:49:20.182+0000 7f0606e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2024-09-10T20:49:20.183+0000 7f0606e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 0 2024-09-10T20:49:24.688 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2024-09-10T20:49:21.546+0000 7f0606e44640 -1 Falling back to public interface 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 1 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.689 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2024-09-10T20:49:24.085+0000 7f0606e44640 -1 osd.3 78 log_to_monitors true 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 3 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers52730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:24.962 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2024-09-10T20:49:20.376+0000 7f905b20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2024-09-10T20:49:20.378+0000 7f905b20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2024-09-10T20:49:20.379+0000 7f905b20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 0 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:24.963 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2024-09-10T20:49:21.267+0000 7f905b20f640 -1 Falling back to public interface 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 1 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2 2024-09-10T20:49:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2024-09-10T20:49:23.356+0000 7f905b20f640 -1 osd.1 78 log_to_monitors true 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 3 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:25.791 INFO:tasks.workunit.client.0.smithi181.stderr:152730: osd.1 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [74,78) [v2:127.0.0.1:6810/3525045333,v1:127.0.0.1:6811/3525045333] [v2:127.0.0.1:6812/3525045333,v1:127.0.0.1:6813/3525045333] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:49:25.791 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:25.791 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:25.792 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 1 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 2 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: 3' 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-10T20:49:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285570 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285570 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-356482285570' 2024-09-10T20:49:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:26.629 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph t.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:26.629 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:26.629 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:26.629 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:49:26.629 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 4 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: osd.3 up in weight 1 up_from 85 up_thru 85 down_at 79 last_clean_interval [60,78) [v2:127.0.0.1:6826/2495373969,v1:127.0.0.1:6827/2495373969] [v2:127.0.0.1:6828/2495373969,v1:127.0.0.1:6829/2495373969] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:26.630 INFO:tasks.workunit.client.0.smithi181.stderr:152728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 1 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 2 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: 3' 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:26.631 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:26.632 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:26.632 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-10T20:49:26.632 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-10T20:49:26.632 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-10T20:49:26.632 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:26.930 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpell osd.2 flush_pg_stats 2024-09-10T20:49:26.930 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514503 2024-09-10T20:49:26.930 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514503 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-356482285570 2-326417514503' 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220162 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220162 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-356482285570 2-326417514503 3-365072220162' 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:26.931 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: waiting osd.0 seq 21474836540 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-356482285570 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-356482285570 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285570 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-09-10T20:49:26.932 INFO:tasks.workunit.client.0.smithi181.stderr:152730: waiting osd.1 seq 356482285570 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285570 -lt 356482285570 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514503 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:26.933 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514503 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephters.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285571 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285571 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-356482285571' 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514504 2024-09-10T20:49:28.613 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514504 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-356482285571 2-326417514504' 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220163 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220163 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-356482285571 2-326417514504 3-365072220163' 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-10T20:49:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: waiting osd.0 seq 21474836542 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836542 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-356482285571 2024-09-10T20:49:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:49:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-356482285571 2024-09-10T20:49:28.616 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514503 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514503' 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: waiting osd.2 seq 326417514503 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514503 -lt 326417514503 2024-09-10T20:49:29.455 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-365072220162 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-365072220162 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220162 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: waiting osd.3 seq 365072220162 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 365072220162 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:29.456 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220163 -lt 365072220162 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:49:29.457 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qauntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285571 2024-09-10T20:49:29.965 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: waiting osd.1 seq 356482285571 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285571 -lt 356482285571 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514504 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514504 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514504 2024-09-10T20:49:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514504' 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: waiting osd.2 seq 326417514504 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514504 -lt 326417514504 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-365072220163 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-365072220163 2024-09-10T20:49:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220163 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: waiting osd.3 seq 365072220163 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220163 -lt 365072220163 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:29.968 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:29.969 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:29.969 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:30.954 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("s/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:49:30.956 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:49:30.957 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:49:30.957 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:49:30.957 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.clietale") | not)' 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:31.360 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:49:31.361 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:32.569 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:152728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:32.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:32.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 152727 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:nt.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:32.673 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:49:32.674 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:32.675 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:33.289 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:33.289 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:33.289 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:33.289 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:152730: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:49:33.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:49:33.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:49:33.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T20:49:33.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T20:49:33.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:49:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:49:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:33.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:49:34.139 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:49:34.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:34.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:34.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:34.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:34.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:34.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:34.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:34.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:35.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:35.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:35.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:35.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:35.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:35.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:35.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:35.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:35.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:36.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:36.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:36.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:36.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:36.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:36.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:36.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:37.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:38.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:38.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:38.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:38.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:38.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:38.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:38.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:38.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:38.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:39.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:39.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:39.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:39.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:39.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:39.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:39.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:39.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:48:55.959479+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:39.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:49:40.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:49:40.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:49:40.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:49:40.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:49:40.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:49:40.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:49:40.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:49:40.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:49:35.068496+0000 '>' 2024-09-10T20:48:55.959479+0000 2024-09-10T20:49:40.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:49:40.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:49:42.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T20:49:42.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:49:42.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:42.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 155948"' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 155948' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 155950"' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 155950' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 155948 155950' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/155951: /' 2024-09-10T20:49:42.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:42.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 155948 2024-09-10T20:49:42.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:42.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/155953: /' 2024-09-10T20:49:43.948 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.948 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:43.948 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:49:43.948 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:49:43.949 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: _ 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: hinfo_key 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: snapset 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:43.950 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:49:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:43.952 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:49:43.953 INFO:tasks.workunit.client.0.smithi181.stderr:155953: start osd.1 2024-09-10T20:49:43.991 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.991 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:43.992 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: _ 2024-09-10T20:49:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:155951: hinfo_key 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: snapset 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:43.994 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:43.995 INFO:tasks.workunit.client.0.smithi181.stderr:155951: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:43.996 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:43.997 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:49:43.997 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:49:43.997 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:49:43.997 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:49:43.997 INFO:tasks.workunit.client.0.smithi181.stderr:155951: start osd.3 2024-09-10T20:49:48.255 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2024-09-10T20:49:44.020+0000 7f8c9040b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2024-09-10T20:49:44.056+0000 7f8c9040b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2024-09-10T20:49:44.057+0000 7f8c9040b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:49:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 0 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 1 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.257 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2024-09-10T20:49:45.666+0000 7f8c9040b640 -1 Falling back to public interface 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2024-09-10T20:49:47.539+0000 7f8c9040b640 -1 osd.3 88 log_to_monitors true 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 3 2024-09-10T20:49:48.258 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.926 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersvery_ops 2024-09-10T20:49:48.926 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2024-09-10T20:49:43.978+0000 7f1e05e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2024-09-10T20:49:44.056+0000 7f1e05e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2024-09-10T20:49:44.058+0000 7f1e05e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:49:48.927 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 0 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 1 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2024-09-10T20:49:45.897+0000 7f1e05e1e640 -1 Falling back to public interface 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 3 2024-09-10T20:49:48.929 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:48.930 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:48.930 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:49:49.652 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2024-09-10T20:49:48.921+0000 7f1e.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: osd.3 up in weight 1 up_from 93 up_thru 93 down_at 89 last_clean_interval [85,88) [v2:127.0.0.1:6810/3441185659,v1:127.0.0.1:6811/3441185659] [v2:127.0.0.1:6812/3441185659,v1:127.0.0.1:6813/3441185659] exists,up 1d4cfff0-7220-4b2c-8a04-0a84a04899a9 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 1 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 2 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: 3' 2024-09-10T20:49:49.654 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:49:49.655 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg05e1e640 -1 osd.1 88 log_to_monitors true 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 4 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:49:50.761 INFO:tasks.workunit.client.0.smithi181.stderr:155953: osd.1 up in weight 1 up_from 97 up_thru 83 down_at 89 last_clean_interval [83,88) [v2:127.0.0.1:6826/1623508392,v1:127.0.0.1:6827/1623508392] [v2:127.0.0.1:6828/1623508392,v1:127.0.0.1:6829/1623508392] exists,up a909f2fa-8cca-463a-8ac4-b3a884be0d01 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:49:50.762 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 1 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 2 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: 3' 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:49:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-10T20:49:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-10T20:49:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-10T20:49:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:50.942 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone_stats: for osd in $ids 2024-09-10T20:49:50.942 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:49:50.942 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514509 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514509 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 2-326417514509' 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958530 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958530 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 2-326417514509 3-399431958530' 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-10T20:49:50.943 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: waiting osd.0 seq 21474836547 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836547 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514509 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514509 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514509 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514509' 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: waiting osd.2 seq 326417514509 2024-09-10T20:49:50.944 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514509 -lt 326417514509 2024-09-10T20:49:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-399431958530 2024-09-10T20:49:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:52.327 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-399431958530 2024-09-10T20:49:52.327 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:52.327 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958530 2024-09-10T20:49:52.327 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-09-10T20:49:52.327 INFO:tasks.workunit.client.0.smithi181.stderr:155951: waiting osd.3 seq 399431958530 2024-09-10T20:49:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958530 -lt 399431958530 2024-09-10T20:49:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:52.328 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:52.329 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:52.330 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:52.330 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:52.330 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:49:52.330 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:52.330 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:52.331 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:52.331 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-10T20:49:52.331 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' -1 2024-09-10T20:49:52.331 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:49:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-10T20:49:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:49:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:49:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:49:52.332 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:52.886 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contai/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827714 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827714 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-416611827714' 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514511 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514511 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-416611827714 2-326417514511' 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958531 2024-09-10T20:49:52.887 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958531 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-416611827714 2-326417514511 3-399431958531' 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: waiting osd.0 seq 21474836548 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836548 2024-09-10T20:49:52.888 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836548 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-416611827714 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-416611827714 2024-09-10T20:49:52.889 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:53.041 INFO:tasks.workunit.client.0.smithi181.stderr:155953ns("stale") | not)' 2024-09-10T20:49:53.041 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:53.041 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:155951: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:49:53.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 155950 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827714 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 416611827714' 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: waiting osd.1 seq 416611827714 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827714 -lt 416611827714 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-326417514511 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-326417514511 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:54.099 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514511 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 326417514511' 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: waiting osd.2 seq 326417514511 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514511 -lt 326417514511 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-399431958531 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-399431958531 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958531 2024-09-10T20:49:54.100 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: waiting osd.3 seq 399431958531 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958531 -lt 399431958531 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:49:54.101 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:49:54.739 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:155953: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:155953: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:49:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:49:54.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:49:54.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:49:54.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:49:54.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:49:54.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:49:54.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:54.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:54.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:54.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:54.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:54.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:54.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:49:54.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:49:54.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:49:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:49:54.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:49:54.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:49:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:49:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:49:54.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:49:54.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:49:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:49:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:49:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:49:54.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:49:54.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:54.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:54.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:49:54.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:49:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:49:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:49:54.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:49:54.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:49:54.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:49:54.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:49:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:49:54.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:49:54.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:49:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:49:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:49:54.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:49:54.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:49:54.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:49:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:49:54.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:49:54.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:49:54.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:54.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:54.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:49:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:49:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:49:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:49:54.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:49:54.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:54.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:49:54.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:49:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:49:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:49:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:49:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:49:54.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:49:55.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:49:55.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:55.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:55.373 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:55.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:55.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:49:55.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:49:55.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:49:55.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:49:55.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:49:55.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:55.568 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:49:55.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:49:55.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:49:55.652 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:49:55.653 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:49:55.653 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:55.653 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:55.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:49:55.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:49:55.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:49:55.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-10T20:49:55.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:49:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:49:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:49:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:49:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:49:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:49:56.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:56.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:49:56.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:49:56.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-10T20:49:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:49:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:49:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:49:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:49:56.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:49:56.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e24dd852-68a1-44b4-a24e-fadbc4114249 2024-09-10T20:49:56.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e24dd852-68a1-44b4-a24e-fadbc4114249' 2024-09-10T20:49:56.578 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 e24dd852-68a1-44b4-a24e-fadbc4114249 2024-09-10T20:49:56.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:49:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0sOBm5twqIxAAWvaGLMXic6HYayJ/Rdv+2Q== 2024-09-10T20:49:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0sOBm5twqIxAAWvaGLMXic6HYayJ/Rdv+2Q=="}' 2024-09-10T20:49:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e24dd852-68a1-44b4-a24e-fadbc4114249 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:49:56.798 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:49:56.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:49:56.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0sOBm5twqIxAAWvaGLMXic6HYayJ/Rdv+2Q== --osd-uuid e24dd852-68a1-44b4-a24e-fadbc4114249 2024-09-10T20:49:56.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:49:56.835+0000 7ff954842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:56.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:49:56.845+0000 7ff954842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:56.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:49:56.846+0000 7ff954842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:49:56.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:49:56.846+0000 7ff954842640 -1 bdev(0x555846869c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:49:56.850 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:49:56.847+0000 7ff954842640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:50:02.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:50:02.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:50:02.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:50:02.332 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:50:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:50:02.692 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:50:02.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:50:02.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:50:02.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:50:02.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:50:02.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:50:02.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:02.716+0000 7efec4345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:02.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:02.735+0000 7efec4345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:02.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:02.736+0000 7efec4345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:50:02.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:02.970 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:50:02.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:50:02.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:02.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:03.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:04.246 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:50:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:50:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:04.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:04.348+0000 7efec4345640 -1 Falling back to public interface 2024-09-10T20:50:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:05.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:05.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:05.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:50:05.528 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:50:05.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:05.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:05.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:06.405 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:06.401+0000 7efec4345640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:50:06.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:06.820 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:50:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:50:06.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:06.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:07.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:08.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:08.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:08.361 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:50:08.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:50:08.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:08.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:08.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:09.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:09.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:09.650 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:50:09.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:50:09.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:09.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:09.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:10.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:10.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:10.969 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:50:10.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:50:10.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:10.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:11.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:12.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:12.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:12.370 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:50:12.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:50:12.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:12.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:12.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:13.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:13.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:13.661 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:50:13.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:50:13.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:13.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:13.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:14.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:14.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:14.986 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:50:14.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:50:14.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:14.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:15.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:16.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:16.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:16.361 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:50:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:50:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:16.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:16.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:17.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:17.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:17.649 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:50:17.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:50:17.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:17.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:18.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:19.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:19.026 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:50:19.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:50:19.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:19.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:20.435 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:50:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:50:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:20.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:21.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:22.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:22.222 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:50:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:50:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:22.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:22.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:23.641 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:50:23.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:50:23.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:23.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:24.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:25.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:25.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:25.229 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:50:25.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:50:25.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:25.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:25.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:26.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:26.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:26.864 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:50:26.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:50:26.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:26.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:27.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:28.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:28.306 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:50:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:50:28.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:28.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:28.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:29.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:29.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:29.642 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:50:29.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:50:29.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:29.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:50:29.922 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1513325215,v1:127.0.0.1:6803/1513325215] [v2:127.0.0.1:6804/1513325215,v1:127.0.0.1:6805/1513325215] exists,up e24dd852-68a1-44b4-a24e-fadbc4114249 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:50:29.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:50:29.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:50:29.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:50:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:50:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:50:29.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd' 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:50:29.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:50:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVseBm8CYROBAAgHynG7dEKfb7HRAL0IiDSQ== 2024-09-10T20:50:29.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVseBm8CYROBAAgHynG7dEKfb7HRAL0IiDSQ=="}' 2024-09-10T20:50:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd -i td/osd-scrub-repair/1/new.json 2024-09-10T20:50:30.301 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:50:30.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:50:30.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVseBm8CYROBAAgHynG7dEKfb7HRAL0IiDSQ== --osd-uuid a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:50:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:30.340+0000 7f24fdcb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:30.374+0000 7f24fdcb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:30.375+0000 7f24fdcb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:30.375+0000 7f24fdcb0640 -1 bdev(0x5580cffe8800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:50:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:30.375+0000 7f24fdcb0640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:50:35.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:50:35.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:50:35.994 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:50:35.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:50:35.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:50:36.361 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:50:36.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:50:36.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:50:36.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:50:36.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:50:36.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:50:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:36.383+0000 7fc045c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:36.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:36.385+0000 7fc045c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:36.391 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:36.387+0000 7fc045c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:36.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:50:36.640 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:50:36.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:36.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:36.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:37.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:37.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:37.934 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:50:37.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:50:37.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:37.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:38.013 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:38.009+0000 7fc045c0d640 -1 Falling back to public interface 2024-09-10T20:50:38.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:39.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:39.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:39.217 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:50:39.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:50:39.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:39.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:39.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:40.084 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:50:40.079+0000 7fc045c0d640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:50:40.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:40.508 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:50:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:50:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:40.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:41.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:41.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:41.861 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:50:41.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:50:41.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:41.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:42.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:43.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:43.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:43.144 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:50:43.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:50:43.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:43.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:43.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:44.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:44.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:44.430 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:50:44.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:50:44.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:44.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:44.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:45.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:45.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:45.727 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:50:45.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:50:45.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:45.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:46.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:47.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:47.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:47.100 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:50:47.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:50:47.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:47.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:47.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:48.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:48.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:48.386 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:50:48.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:50:48.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:48.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:48.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:49.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:50:49.678 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:50:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:49.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:49.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:50.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:50.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:50.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:50:50.962 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:50:50.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:50.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:51.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:52.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:52.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:52.243 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:50:52.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:50:52.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:52.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:52.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:53.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:53.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:53.533 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:50:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:50:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:53.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:54.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:55.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:55.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:55.155 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:50:55.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:50:55.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:55.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:55.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:56.573 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:50:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:50:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:57.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:58.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:58.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:58.016 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:50:58.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:50:58.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:58.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:50:58.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:50:59.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:50:59.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:50:59.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:50:59.535 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:50:59.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:50:59.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:51:00.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:01.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:01.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:01.196 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:51:01.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:51:01.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:01.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:51:01.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:02.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:02.982 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:51:02.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:02.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:51:02.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:02.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:51:03.293 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/134598118,v1:127.0.0.1:6811/134598118] [v2:127.0.0.1:6812/134598118,v1:127.0.0.1:6813/134598118] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:51:03.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:51:03.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:51:03.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:51:03.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:51:03.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:51:03.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:51:03.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:51:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:51:03.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 70ec98f1-9bdd-4005-a3f1-b36c05db444f' 2024-09-10T20:51:03.305 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:51:03.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:51:03.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3seBmZpubEhAA6wSBYi773HOBl/lWbQ09KQ== 2024-09-10T20:51:03.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3seBmZpubEhAA6wSBYi773HOBl/lWbQ09KQ=="}' 2024-09-10T20:51:03.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 70ec98f1-9bdd-4005-a3f1-b36c05db444f -i td/osd-scrub-repair/2/new.json 2024-09-10T20:51:03.667 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:51:03.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:51:03.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3seBmZpubEhAA6wSBYi773HOBl/lWbQ09KQ== --osd-uuid 70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:51:03.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:03.705+0000 7fab00839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:03.751 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:03.747+0000 7fab00839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:03.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:03.748+0000 7fab00839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:03.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:03.749+0000 7fab00839640 -1 bdev(0x5565140f7c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:51:03.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:03.749+0000 7fab00839640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:51:10.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:51:10.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:51:10.519 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:51:10.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:51:10.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:51:10.880 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:51:10.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:51:10.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:51:10.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:51:10.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:51:10.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:51:10.907 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:10.904+0000 7fbfe7824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:10.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:10.906+0000 7fbfe7824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:10.910 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:10.907+0000 7fbfe7824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:11.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:51:11.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:51:11.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:11.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:11.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:12.438 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:51:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:51:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:12.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:12.514+0000 7fbfe7824640 -1 Falling back to public interface 2024-09-10T20:51:12.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:13.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:13.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:13.735 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:51:13.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:51:13.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:13.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:14.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:14.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:14.665+0000 7fbfe7824640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:51:15.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:15.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:15.025 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:51:15.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:51:15.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:15.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:15.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:16.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:16.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:16.461 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:51:16.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:51:16.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:16.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:16.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:17.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:51:17.845 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:51:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:17.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:18.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:19.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:19.139 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:51:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:51:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:19.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:20.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:20.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:20.426 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:51:20.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:51:20.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:20.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:20.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:21.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:21.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:21.881 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:51:21.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:51:21.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:21.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:22.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:23.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:23.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:23.167 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:51:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:51:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:23.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:23.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:24.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:24.467 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:51:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:51:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:24.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:24.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:25.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:25.793 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:51:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:51:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:25.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:26.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:27.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:27.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:27.080 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:51:27.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:51:27.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:27.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:27.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:28.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:28.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:28.421 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:51:28.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:51:28.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:28.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:28.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:29.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:29.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:29.890 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:51:29.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:51:29.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:29.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:30.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:31.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:31.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:31.203 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:51:31.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:51:31.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:31.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:31.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:32.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:32.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:32.919 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:51:32.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:51:32.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:32.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:33.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:34.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:34.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:34.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:51:34.453 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:51:34.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:34.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:35.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:36.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:36.416 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:51:36.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:36.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:51:36.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:36.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:36.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:37.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:37.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:37.835 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:51:37.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:51:37.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:37.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:38.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:39.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:39.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:39.126 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T20:51:39.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T20:51:39.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:39.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:51:39.411 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4145627431,v1:127.0.0.1:6819/4145627431] [v2:127.0.0.1:6820/4145627431,v1:127.0.0.1:6821/4145627431] exists,up 70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:51:39.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:51:39.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:51:39.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:51:39.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:51:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:51:39.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:51:39.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 c55c48f2-4dba-4d29-92ad-cad6f1ab8b78' 2024-09-10T20:51:39.419 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:51:39.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:51:39.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBbseBmivqfGRAArsv7qrnV9v4XQ/WjE75/kg== 2024-09-10T20:51:39.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBbseBmivqfGRAArsv7qrnV9v4XQ/WjE75/kg=="}' 2024-09-10T20:51:39.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 -i td/osd-scrub-repair/3/new.json 2024-09-10T20:51:39.796 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:51:39.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T20:51:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbseBmivqfGRAArsv7qrnV9v4XQ/WjE75/kg== --osd-uuid c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:51:39.837 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:39.833+0000 7f342c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:39.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:39.876+0000 7f342c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:39.877+0000 7f342c41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:39.878+0000 7f342c41e640 -1 bdev(0x557428917c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:51:39.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:39.878+0000 7f342c41e640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T20:51:45.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T20:51:45.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:51:45.956 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T20:51:45.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:51:45.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:51:46.318 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T20:51:46.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:51:46.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:51:46.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:51:46.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:51:46.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:51:46.346 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:46.342+0000 7f5036a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:46.348 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:46.344+0000 7f5036a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:46.345+0000 7f5036a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:51:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:51:46.663 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:51:46.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:46.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:46.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:47.700 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:47.696+0000 7f5036a13640 -1 Falling back to public interface 2024-09-10T20:51:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:47.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:51:47.938 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:51:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:47.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:48.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:49.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:49.222 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:51:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:51:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:49.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:49.577 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:51:49.573+0000 7f5036a13640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:51:50.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:50.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:50.510 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:51:50.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:51:50.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:50.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:50.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:51.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:51.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:51.886 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:51:51.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:51:51.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:51.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:52.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:53.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:53.179 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:51:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:51:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:53.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:53.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:54.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:54.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:54.470 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:51:54.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:51:54.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:54.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:54.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:55.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:55.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:55.824 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:51:55.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:51:55.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:55.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:56.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:57.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:57.117 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:51:57.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:57.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:51:57.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:57.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:57.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:58.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:58.413 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:51:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:51:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:58.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:51:58.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:51:59.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:51:59.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:51:59.725 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:51:59.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:51:59.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:51:59.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:00.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:01.013 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:52:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:52:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:01.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:02.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:02.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:02.398 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:52:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:52:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:02.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:03.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:04.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:04.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:04.208 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:52:04.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:52:04.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:04.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:04.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:05.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:05.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:05.825 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:52:05.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:52:05.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:05.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:06.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:07.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:07.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:07.584 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:52:07.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:52:07.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:07.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:07.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:08.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:08.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:08.914 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:52:08.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:52:08.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:08.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:09.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:10.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:10.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:10.494 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:52:10.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:52:10.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:10.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:11.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:12.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:12.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:12.086 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:52:12.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:52:12.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:12.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:13.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:13.422 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:52:13.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:52:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:13.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:13.698 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1304381989,v1:127.0.0.1:6827/1304381989] [v2:127.0.0.1:6828/1304381989,v1:127.0.0.1:6829/1304381989] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:52:13.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:52:13.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:52:13.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:52:13.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-10T20:52:13.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T20:52:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T20:52:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T20:52:13.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T20:52:14.407 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T20:52:14.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:52:15.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T20:52:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-10T20:52:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:52:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:52:17.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:52:17.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:52:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:52:17.850 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:52:17.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:52:17.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:52:17.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:52:17.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:52:18.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:52:18.227 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:18.227 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:18.227 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:52:18.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:52:18.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:18.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:52:18.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T20:52:18.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T20:52:18.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T20:52:18.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:18.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:52:18.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-10T20:52:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-10T20:52:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672981' 2024-09-10T20:52:18.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:18.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:52:18.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-10T20:52:18.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-10T20:52:18.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509453' 2024-09-10T20:52:18.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:18.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:52:18.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345924 2024-09-10T20:52:18.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345924 2024-09-10T20:52:18.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509453 3-85899345924' 2024-09-10T20:52:18.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:18.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T20:52:18.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:18.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:52:18.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T20:52:18.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:18.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T20:52:18.643 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836503 2024-09-10T20:52:18.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T20:52:18.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:18.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T20:52:18.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:18.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-10T20:52:18.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:18.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:52:18.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-10T20:52:18.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:18.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-10T20:52:18.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-10T20:52:18.957 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672981 2024-09-10T20:52:18.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:52:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-10T20:52:19.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:19.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-10T20:52:19.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:19.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:52:19.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-10T20:52:19.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-10T20:52:19.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-10T20:52:19.235 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509453 2024-09-10T20:52:19.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:52:19.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509453 2024-09-10T20:52:19.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:52:20.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:52:20.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:52:20.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-10T20:52:20.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:20.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345924 2024-09-10T20:52:20.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:20.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:52:20.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345924 2024-09-10T20:52:20.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:20.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345924 2024-09-10T20:52:20.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2024-09-10T20:52:20.802 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345924 2024-09-10T20:52:20.802 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:52:21.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345924 -lt 85899345924 2024-09-10T20:52:21.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:52:21.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:21.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:52:21.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:52:21.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:52:21.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:52:21.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:52:21.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:52:21.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:21.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:22.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T20:52:22.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:52:22.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:52:22.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-09-10T20:52:22.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T20:52:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T20:52:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T20:52:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T20:52:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-10T20:52:22.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T20:52:22.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T20:52:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T20:52:23.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:52:24.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T20:52:24.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T20:52:24.649 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 2 allow_ec_overwrites to true 2024-09-10T20:52:24.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T20:52:24.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:52:24.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:52:24.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:52:24.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:52:24.662 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:52:24.662 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:52:24.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:52:24.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:52:24.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:52:24.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:52:24.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:52:24.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:52:24.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:52:24.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:52:24.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:52:25.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:52:25.035 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:25.035 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:25.035 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:52:25.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:52:25.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:25.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:52:25.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T20:52:25.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T20:52:25.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T20:52:25.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:25.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:52:25.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-10T20:52:25.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-10T20:52:25.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672984' 2024-09-10T20:52:25.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:25.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:52:25.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-10T20:52:25.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-10T20:52:25.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509456' 2024-09-10T20:52:25.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:25.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:52:25.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-10T20:52:25.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-10T20:52:25.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509456 3-85899345926' 2024-09-10T20:52:25.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:25.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T20:52:25.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:52:25.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T20:52:25.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T20:52:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T20:52:25.448 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836506 2024-09-10T20:52:25.448 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:25.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-10T20:52:25.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:52:26.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:52:26.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:27.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T20:52:27.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:27.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-10T20:52:27.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:27.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:52:27.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-10T20:52:27.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:27.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-10T20:52:27.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-10T20:52:27.005 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672984 2024-09-10T20:52:27.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:52:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-10T20:52:27.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:27.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-10T20:52:27.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:52:27.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-10T20:52:27.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-10T20:52:27.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-10T20:52:27.283 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509456 2024-09-10T20:52:27.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:52:27.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-10T20:52:27.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:27.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-10T20:52:27.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:27.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:52:27.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-10T20:52:27.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:27.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-10T20:52:27.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-10T20:52:27.561 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345926 2024-09-10T20:52:27.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:52:27.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-10T20:52:27.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:52:27.897 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:27.897 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:28.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:52:28.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:52:28.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:52:28.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:52:28.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:52:28.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:28.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:52:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:52:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:52:28.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T20:52:28.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T20:52:29.233 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T20:52:29.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T20:52:29.760 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T20:52:29.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T20:52:29.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T20:52:29.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T20:52:30.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T20:52:30.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T20:52:30.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T20:52:30.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:52:30.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:52:30.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-10T20:52:30.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T20:52:30.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-10T20:52:30.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T20:52:30.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T20:52:30.297 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:52:30.297 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T20:52:30.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T20:52:30.580 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:30.580 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:30.580 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T20:52:30.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:52:30.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:52:30.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:52:30.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:52:30.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:52:30.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:52:30.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:52:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:52:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:52:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:52:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T20:52:32.998 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T20:52:33.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:52:33.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:52:33.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:52:33.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:52:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:52:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:52:33.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:33.543+0000 7fb5bca0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:33.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:33.545+0000 7fb5bca0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:33.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:33.546+0000 7fb5bca0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:52:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:52:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:52:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:33.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:34.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:34.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:34.419+0000 7fb5bca0c640 -1 Falling back to public interface 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:35.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:35.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:35.874 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:35.870+0000 7fb5bca0c640 -1 osd.3 38 log_to_monitors true 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:36.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/633668504,v1:127.0.0.1:6827/633668504] [v2:127.0.0.1:6828/633668504,v1:127.0.0.1:6829/633668504] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:52:36.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:52:36.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:52:36.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:52:37.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:52:37.128 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:37.128 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:37.129 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:52:37.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:52:37.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:37.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:52:37.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-10T20:52:37.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-10T20:52:37.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-10T20:52:37.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:37.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:52:37.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-10T20:52:37.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-10T20:52:37.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672987' 2024-09-10T20:52:37.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:37.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:52:37.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-10T20:52:37.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-10T20:52:37.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-64424509459' 2024-09-10T20:52:37.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:37.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:52:37.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-10T20:52:37.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-10T20:52:37.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-64424509459 3-184683593730' 2024-09-10T20:52:37.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:37.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-10T20:52:37.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:37.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:52:37.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-10T20:52:37.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:37.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-10T20:52:37.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-10T20:52:37.749 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836509 2024-09-10T20:52:37.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:38.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-10T20:52:38.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:52:39.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:52:39.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:39.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-10T20:52:39.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:39.311 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-10T20:52:39.311 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:39.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:52:39.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-10T20:52:39.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:39.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-10T20:52:39.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-10T20:52:39.314 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672987 2024-09-10T20:52:39.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:52:39.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-10T20:52:39.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:39.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-10T20:52:39.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:39.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:52:39.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-10T20:52:39.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-10T20:52:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-10T20:52:39.592 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509459 2024-09-10T20:52:39.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:52:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-10T20:52:39.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:39.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-10T20:52:39.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:39.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:52:39.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-10T20:52:39.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:39.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-10T20:52:39.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-10T20:52:39.874 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 184683593730 2024-09-10T20:52:39.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:52:40.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-10T20:52:40.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:52:40.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:40.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:40.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:52:40.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:52:40.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:52:40.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:52:40.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:40.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:41.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:52:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:52:41.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T20:52:41.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T20:52:41.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:52:41.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:52:41.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:41.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:41.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:41.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:41.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:41.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:52:41.953 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:52:41.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:41.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:41.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:41.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:42.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:23.052338+0000 '>' 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:42.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:52:43.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:52:43.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:52:43.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:52:43.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:43.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:43.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:43.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:43.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:23.052338+0000 '>' 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:43.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:52:44.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:52:44.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:52:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:52:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:44.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:23.052338+0000 '>' 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:44.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:52:45.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:52:45.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:52:45.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:52:45.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:45.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:45.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:45.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:46.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:23.052338+0000 '>' 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:46.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:52:47.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:52:47.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:52:47.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:52:47.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:52:47.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:52:47.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:52:47.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:52:47.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:42.913033+0000 '>' 2024-09-10T20:52:23.052338+0000 2024-09-10T20:52:47.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:52:47.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:52:49.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:52:49.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:52:49.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:52:49.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:52:49.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:52:49.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:52:50.185 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:52:50.185 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:52:50.186 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:52:50.186 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:52:50.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:52:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:52:50.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:52:50.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:52:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:52:50.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:52:50.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:52:50.485 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:50.481+0000 7fdc3ca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:50.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:50.557+0000 7fdc3ca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:50.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:50.559+0000 7fdc3ca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:52:50.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:50.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:51.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:51.693 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:51.689+0000 7fdc3ca10640 -1 Falling back to public interface 2024-09-10T20:52:52.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:52:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:52.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:52.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:53.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:53.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:53.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:52:53.413 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:53.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:52:54.066 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:52:54.063+0000 7fdc3ca10640 -1 osd.3 45 log_to_monitors true 2024-09-10T20:52:54.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:52:54.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:52:54.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:52:54.703 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:52:54.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:52:54.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:52:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/1022723354,v1:127.0.0.1:6827/1022723354] [v2:127.0.0.1:6828/1022723354,v1:127.0.0.1:6829/1022723354] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:52:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:52:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:52:55.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:52:55.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:52:55.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:52:55.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:52:55.480 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:52:55.480 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:52:55.480 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:52:55.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:52:55.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:55.481 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:52:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-10T20:52:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-10T20:52:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-10T20:52:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:55.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:52:55.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672992 2024-09-10T20:52:55.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672992 2024-09-10T20:52:55.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672992' 2024-09-10T20:52:55.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:55.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:52:55.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-10T20:52:55.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-10T20:52:55.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-64424509464' 2024-09-10T20:52:55.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:52:55.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:52:55.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-64424509464 3-214748364802' 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:55.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:52:55.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-10T20:52:55.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:55.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-10T20:52:55.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-10T20:52:55.980 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836514 2024-09-10T20:52:55.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836514 2024-09-10T20:52:56.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:52:57.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:52:57.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:52:57.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-10T20:52:57.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:57.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672992 2024-09-10T20:52:57.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:57.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:52:57.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672992 2024-09-10T20:52:57.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:57.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672992 2024-09-10T20:52:57.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-10T20:52:57.548 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672992 2024-09-10T20:52:57.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:52:57.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-10T20:52:57.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:57.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-10T20:52:57.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:57.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:52:57.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-10T20:52:57.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-10T20:52:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-10T20:52:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509464 2024-09-10T20:52:57.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:52:58.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-10T20:52:58.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:52:58.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-10T20:52:58.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:52:58.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:52:58.105 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-10T20:52:58.105 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:52:58.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-10T20:52:58.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-10T20:52:58.107 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 214748364802 2024-09-10T20:52:58.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:52:58.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-10T20:52:58.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:52:58.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:58.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:58.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:52:58.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:52:58.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:52:59.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:52:59.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:52:59.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:59.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:52:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:52:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T20:52:59.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:52:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:52:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:52:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:52:59.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:52:59.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:52:59.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:52:59.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:52:59.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:53:00.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:53:00.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:53:00.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:53:00.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:53:00.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:53:00.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:00.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:00.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:53:00.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:00.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:00.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:01.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:53:01.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:53:01.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:53:01.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:53:01.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:01.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:01.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:01.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:53:03.709 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:04.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:04.235 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:04.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:53:04.237 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:04.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:04.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:04.262+0000 7f008881f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:04.268 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:04.264+0000 7f008881f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:04.269 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:04.266+0000 7f008881f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:04.525 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:53:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:04.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:05.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:05.631+0000 7f008881f640 -1 Falling back to public interface 2024-09-10T20:53:05.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:53:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:05.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:07.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:53:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:07.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:07.287 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:07.284+0000 7f008881f640 -1 osd.1 52 log_to_monitors true 2024-09-10T20:53:07.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:08.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/147945973,v1:127.0.0.1:6811/147945973] [v2:127.0.0.1:6812/147945973,v1:127.0.0.1:6813/147945973] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:08.775 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:08.776 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:08.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:08.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:08.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:08.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:09.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:09.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-10T20:53:09.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-10T20:53:09.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-10T20:53:09.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:09.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:53:09.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-10T20:53:09.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-10T20:53:09.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-244813135874' 2024-09-10T20:53:09.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:09.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:53:09.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509468 2024-09-10T20:53:09.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509468 2024-09-10T20:53:09.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-244813135874 2-64424509468' 2024-09-10T20:53:09.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:09.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-244813135874 2-64424509468 3-214748364806' 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-10T20:53:09.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:09.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:53:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-10T20:53:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:09.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-10T20:53:09.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-10T20:53:09.607 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836517 2024-09-10T20:53:09.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-10T20:53:09.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:53:10.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:53:10.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:11.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836517 2024-09-10T20:53:11.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:11.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-10T20:53:11.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:11.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:53:11.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-10T20:53:11.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:11.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-10T20:53:11.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-10T20:53:11.173 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135874 2024-09-10T20:53:11.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:53:11.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-10T20:53:11.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:11.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509468 2024-09-10T20:53:11.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:11.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:53:11.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509468 2024-09-10T20:53:11.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:11.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509468 2024-09-10T20:53:11.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-10T20:53:11.457 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509468 2024-09-10T20:53:11.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:53:11.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-10T20:53:11.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:11.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-10T20:53:11.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:11.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:53:11.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-10T20:53:11.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:11.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-10T20:53:11.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-10T20:53:11.741 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 214748364806 2024-09-10T20:53:11.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:53:12.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-10T20:53:12.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:53:12.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:12.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:12.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:53:12.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:12.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:12.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:53:12.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:12.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:12.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:53:12.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:13.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:13.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:52:42.913033+0000 2024-09-10T20:53:13.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:53:13.813 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:53:13.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:52:42.913033+0000 2024-09-10T20:53:13.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:52:42.913033+0000 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:14.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:52:42.913033+0000 '>' 2024-09-10T20:52:42.913033+0000 2024-09-10T20:53:14.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:15.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:15.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:15.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:15.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:15.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:15.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:15.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:15.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:52:42.913033+0000 2024-09-10T20:53:15.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:53:15.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:53:17.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:17.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:17.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:17.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:53:17.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:17.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:17.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:53:17.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:17.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:53:17.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:53:18.180 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:53:18.181 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T20:53:18.181 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T20:53:18.181 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:18.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:18.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:53:18.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:18.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:18.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:18.484 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:18.479+0000 7f2a8b018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:18.561 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:18.556+0000 7f2a8b018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:18.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:18.557+0000 7f2a8b018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:18.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:18.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:53:19.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:20.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:20.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:20.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:20.062 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:53:20.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:20.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:53:20.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:20.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:20.391+0000 7f2a8b018640 -1 Falling back to public interface 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:21.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:53:21.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:22.266 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:53:22.263+0000 7f2a8b018640 -1 osd.3 59 log_to_monitors true 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:22.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:53:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 64 up_thru 57 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/2486647643,v1:127.0.0.1:6827/2486647643] [v2:127.0.0.1:6828/2486647643,v1:127.0.0.1:6829/2486647643] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:53:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:22.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:23.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:23.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:23.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:23.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:23.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:23.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:23.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-10T20:53:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-10T20:53:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-10T20:53:23.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:23.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:53:23.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135878 2024-09-10T20:53:23.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135878 2024-09-10T20:53:23.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135878' 2024-09-10T20:53:23.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:23.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:53:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-10T20:53:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-10T20:53:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135878 2-64424509471' 2024-09-10T20:53:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:23.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:53:24.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-10T20:53:24.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-10T20:53:24.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135878 2-64424509471 3-274877906946' 2024-09-10T20:53:24.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:24.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-10T20:53:24.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:24.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:53:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-10T20:53:24.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-10T20:53:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-10T20:53:24.032 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836521 2024-09-10T20:53:24.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:24.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836521 2024-09-10T20:53:24.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:53:25.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:53:25.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:25.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-10T20:53:25.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:25.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135878 2024-09-10T20:53:25.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:25.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:53:25.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135878 2024-09-10T20:53:25.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135878 2024-09-10T20:53:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-09-10T20:53:25.592 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135878 2024-09-10T20:53:25.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:53:25.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135878 -lt 244813135878 2024-09-10T20:53:25.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:25.869 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-10T20:53:25.869 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:25.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:53:25.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-10T20:53:25.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:25.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-10T20:53:25.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-10T20:53:25.872 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509471 2024-09-10T20:53:25.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:53:26.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509471 2024-09-10T20:53:26.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:26.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906946 2024-09-10T20:53:26.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:26.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:53:26.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906946 2024-09-10T20:53:26.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:26.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-10T20:53:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-10T20:53:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 274877906946 2024-09-10T20:53:26.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:53:26.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-10T20:53:26.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:53:26.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:26.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:26.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:27.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:53:27.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:27.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:27.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:27.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:53:27.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:53:27.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:53:27.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:53:27.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:53:27.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-10T20:53:27.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T20:53:27.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 172074"' 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 172074' 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 172075"' 2024-09-10T20:53:27.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 172075' 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 172074 172075' 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 172074 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/172077: /' 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:53:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/172079: /' 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:53:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:30.343 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:30.344 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: start osd.1 2024-09-10T20:53:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:53:30.547 INFO:tasks.workunit.client.0.smithi181.stderr:1172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T20:53:30.548 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-10T20:53:30.549 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:30.550 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:30.551 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: start osd.2 2024-09-10T20:53:30.552 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2024-09-10T20:53:30.370+0000 7f6347c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2024-09-10T20:53:30.372+0000 7f6347c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2024-09-10T20:53:30.373+0000 7f6347c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:53:35.008 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 0 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 1 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:35.009 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2024-09-10T20:53:31.983+0000 7f6347c1e640 -1 Falling back to public interface 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2024-09-10T20:53:34.446+0000 7f6347c1e640 -1 osd.1 66 log_to_monitors true 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 3 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.010 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:35.199 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers72079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T20:53:35.199 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:53:35.199 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2024-09-10T20:53:30.573+0000 7fc6ba80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2024-09-10T20:53:30.576+0000 7fc6ba80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2024-09-10T20:53:30.577+0000 7fc6ba80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 0 2024-09-10T20:53:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2024-09-10T20:53:31.927+0000 7fc6ba80c640 -1 Falling back to public interface 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 1 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2024-09-10T20:53:33.296+0000 7fc6ba80c640 -1 osd.2 66 log_to_monitors true 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 3 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:35.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: osd.2 up in weight 1 up_from 71 up_thru 71 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6810/815778316,v1:127.0.0.1:6811/815778316] [v2:127.0.0.1:6812/815778316,v1:127.0.0.1:6813/815778316] exists,up 70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:36.277 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:36.278 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 1 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 2 2024-09-10T20:53:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:172079: 3' 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836525 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836525 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525' 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:53:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=2 2024-09-10T20:53:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 2 2024-09-10T20:53:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-2' 2024-09-10T20:53:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:36.281 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:53:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /hom.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:53:36.957 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 4 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: osd.1 up in weight 1 up_from 75 up_thru 60 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6818/1608556289,v1:127.0.0.1:6819/1608556289] [v2:127.0.0.1:6820/1608556289,v1:127.0.0.1:6821/1608556289] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:36.958 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 1 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 2 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: 3' 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836526 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836526 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526' 2024-09-10T20:53:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:38.348 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678018 2024-09-10T20:53:38.348 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678018 2024-09-10T20:53:38.348 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-2 2-304942678018' 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906949 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906949 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-2 2-304942678018 3-274877906949' 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836525 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836525 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836525 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-10T20:53:38.349 INFO:tasks.workunit.client.0.smithi181.stderr:172079: waiting osd.0 seq 21474836525 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836525 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-2 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-2 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=2 2024-09-10T20:53:38.350 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-09-10T20:53:38.351 INFO:tasks.workunit.client.0.smithi181.stderr:172079: waiting osd.1 seq 2 2024-09-10T20:53:38.351 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:53:38.351 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 2 -lt 2 2024-09-10T20:53:38.351 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:38.351 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-304942678018 2024-09-10T20:53:39.042 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/ers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:53:39.042 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547203 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547203 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-322122547203' 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678019 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678019 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-322122547203 2-304942678019' 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906950 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906950 2024-09-10T20:53:39.043 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-322122547203 2-304942678019 3-274877906950' 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836526 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836526 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836526 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: waiting osd.0 seq 21474836526 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836525 -lt 21474836526 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:53:39.044 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-322122547203 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-322122547203 2024-09-10T20:53:39.045 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:39.648 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:39.648 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:53:39.648 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-304942678018 2024-09-10T20:53:39.648 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:39.648 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678018 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: waiting osd.2 seq 304942678018 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678019 -lt 304942678018 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906949 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906949 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906949 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906949' 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: waiting osd.3 seq 274877906949 2024-09-10T20:53:39.649 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906949 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:53:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:39.651 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/stanuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547203 2024-09-10T20:53:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-09-10T20:53:40.303 INFO:tasks.workunit.client.0.smithi181.stderr:172077: waiting osd.1 seq 322122547203 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547203 -lt 322122547203 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-304942678019 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-304942678019 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678019 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 304942678019' 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: waiting osd.2 seq 304942678019 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:53:40.304 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678019 -lt 304942678019 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906950 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906950 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906950 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: waiting osd.3 seq 274877906950 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906950 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:53:40.305 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:40.306 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:40.952 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:172077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:53:40.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 172075 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:dalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:41.201 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:53:41.202 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:53:41.203 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:53:42.014 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:53:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:172079: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:172079: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:53:42.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:53:42.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:42.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:42.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:42.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:42.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:53:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:53:42.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:42.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:42.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:43.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:43.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:44.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:44.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:44.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:44.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:44.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:45.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:45.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:45.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:45.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:46.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:46.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:46.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:46.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:46.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:46.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:46.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:46.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:46.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:48.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:14.005016+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:48.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:53:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:53:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:53:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:53:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:53:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:53:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:53:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:53:49.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:14.005016+0000 2024-09-10T20:53:49.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:53:49.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:53:51.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T20:53:51.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:53:51.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 175102"' 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 175102' 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 175103"' 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 175103' 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 175102 175103' 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:53:51.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 175102 2024-09-10T20:53:51.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/175105: /' 2024-09-10T20:53:51.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:53:51.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/175107: /' 2024-09-10T20:53:52.651 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.651 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:52.651 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:53:52.651 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:52.652 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: _ 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: hinfo_key 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: snapset 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.653 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:53:52.654 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:53:52.656 INFO:tasks.workunit.client.0.smithi181.stderr:175105: start osd.1 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T20:53:52.700 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T20:53:52.701 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: _ 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: hinfo_key 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: snapset 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:53:52.702 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:53:52.703 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:53:52.704 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:53:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:53:52.708 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:53:52.708 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:53:52.708 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:53:52.708 INFO:tasks.workunit.client.0.smithi181.stderr:175107: start osd.2 2024-09-10T20:53:57.390 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:53:57.390 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:53:57.390 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:53:57.390 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:57.390 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2024-09-10T20:53:52.681+0000 7f7b52329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2024-09-10T20:53:52.684+0000 7f7b52329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2024-09-10T20:53:52.685+0000 7f7b52329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 0 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:57.391 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2024-09-10T20:53:54.291+0000 7f7b52329640 -1 Falling back to public interface 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 1 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2024-09-10T20:53:56.673+0000 7f7b52329640 -1 osd.1 77 log_to_monitors true 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 3 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.393 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:53:57.661 INFO:tasks.workunit.client.0.smithi181.stderr:175105: osd.1 up in weight 1 up_from 82 up_thru 75 down_atvery_ops 2024-09-10T20:53:57.661 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T20:53:57.661 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:53:57.661 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:53:57.661 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2024-09-10T20:53:52.728+0000 7f697501d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2024-09-10T20:53:52.869+0000 7f697501d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2024-09-10T20:53:52.870+0000 7f697501d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 0 2024-09-10T20:53:57.662 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2024-09-10T20:53:53.983+0000 7f697501d640 -1 Falling back to public interface 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 1 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2024-09-10T20:53:57.082+0000 7f697501d640 -1 osd.2 77 log_to_monitors true 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 3 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:58.287 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ce 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/231896070,v1:127.0.0.1:6819/231896070] [v2:127.0.0.1:6820/231896070,v1:127.0.0.1:6821/231896070] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 1 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 2 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: 3' 2024-09-10T20:53:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836531 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836531 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531' 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-352187318274' 2024-09-10T20:53:58.290 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:53:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285570 2024-09-10T20:53:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285570 2024-09-10T20:53:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-352187318274 2-356482285570' 2024-09-10T20:53:59.476 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906956 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906956 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-352187318274 2-356482285570 3-274877906956' 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836531 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836531 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836531 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-09-10T20:53:59.477 INFO:tasks.workunit.client.0.smithi181.stderr:175105: waiting osd.0 seq 21474836531 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836531 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-352187318274 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-352187318274 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: waiting osd.1 seq 352187318274 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-10T20:53:59.478 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:53:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285570 2024-09-10T20:53:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:53:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:53:59.479 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285570 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntuph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 4 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: osd.2 up in weight 1 up_from 83 up_thru 71 down_at 78 last_clean_interval [71,77) [v2:127.0.0.1:6810/3707940028,v1:127.0.0.1:6811/3707940028] [v2:127.0.0.1:6812/3707940028,v1:127.0.0.1:6813/3707940028] exists,up 70ec98f1-9bdd-4005-a3f1-b36c05db444f 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:53:59.638 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:53:59.639 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 1 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 2 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: 3' 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836532 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836532 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532' 2024-09-10T20:53:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:54:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318275 2024-09-10T20:54:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318275 2024-09-10T20:54:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-352187318275' 2024-09-10T20:54:01.542 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285571 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285571 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-352187318275 2-356482285571' 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-352187318275 2-356482285571 3-274877906957' 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836532 2024-09-10T20:54:01.543 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836532 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836532 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: waiting osd.0 seq 21474836532 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836532 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836532 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-352187318275 2024-09-10T20:54:01.545 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:01.545 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:54:01.545 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-352187318275 2024-09-10T20:54:01.545 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175107/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285570 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285570' 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: waiting osd.2 seq 356482285570 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285570 -lt 356482285570 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:02.014 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906956 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906956 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906956 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: waiting osd.3 seq 274877906956 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906955 -lt 274877906956 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906957 -lt 274877906956 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:02.016 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:02.017 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:02.017 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:02.017 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:02.017 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:02.369 INFO:tasks.workunit.client.0.smithi181.stderr:175105: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:175105: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:54:02.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 175103 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318275 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: waiting osd.1 seq 352187318275 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285571 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:02.865 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285571 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285571 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285571' 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: waiting osd.2 seq 356482285571 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285571 -lt 356482285571 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906957 2024-09-10T20:54:02.866 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906957 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: waiting osd.3 seq 274877906957 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906957 -lt 274877906957 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:02.867 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:02.868 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:03.483 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:175107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:03.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:54:03.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:54:03.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:54:03.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T20:54:03.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:54:03.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:54:03.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:54:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-10T20:54:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T20:54:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T20:54:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-10T20:54:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 177443"' 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 177443' 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 177444"' 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 177444' 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T20:54:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 177443 177444' 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 177443 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/177446: /' 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:54:03.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/177448: /' 2024-09-10T20:54:06.922 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:54:06.922 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:54:06.922 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:06.923 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:54:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:54:06.925 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:54:06.925 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:54:06.925 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:177446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:54:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:177446: start osd.3 2024-09-10T20:54:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:54:07.287 INFO:tasks.workunit.client.0.smithi181.stderr:1177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:54:07.287 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:07.288 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:54:07.289 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:54:07.290 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:54:07.291 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: start osd.1 2024-09-10T20:54:07.292 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:54:11.427 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2024-09-10T20:54:06.951+0000 7f62ad819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2024-09-10T20:54:06.954+0000 7f62ad819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2024-09-10T20:54:06.955+0000 7f62ad819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:54:11.428 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 0 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2024-09-10T20:54:08.069+0000 7f62ad819640 -1 Falling back to public interface 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:54:11.429 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 1 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2024-09-10T20:54:09.763+0000 7f62ad819640 -1 osd.3 85 log_to_monitors true 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.430 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:54:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 3 2024-09-10T20:54:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.431 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177446: osd.3 up in weight 1 up_from 90 up_thru 90 down_at 86 last_c77448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2024-09-10T20:54:07.314+0000 7f1ba9642640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2024-09-10T20:54:07.349+0000 7f1ba9642640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2024-09-10T20:54:07.350+0000 7f1ba9642640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:54:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 0 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2024-09-10T20:54:08.704+0000 7f1ba9642640 -1 Falling back to public interface 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 1 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2024-09-10T20:54:11.016+0000 7f1ba9642640 -1 osd.1 85 log_to_monitors true 2024-09-10T20:54:11.924 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:54:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 3 2024-09-10T20:54:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177448: osd.1 up in weight 1 up_from 93 up_thru 83 down_at 86 last_clean_interval [64,85) [v2:127.0.0.1:6818/1956154959,v1:127.0.0.1:6819/1956154959] [v2:127.0.0.1:6820/1956154959,v1:127.0.0.1:6821/1956154959] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:54:12.437 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:54:12.438 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 1 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 2 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: 3' 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958530 2024-09-10T20:54:12.439 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958530 2024-09-10T20:54:12.440 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-399431958530' 2024-09-10T20:54:12.440 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:12.835 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tlean_interval [82,85) [v2:127.0.0.1:6826/2749901617,v1:127.0.0.1:6827/2749901617] [v2:127.0.0.1:6828/2749901617,v1:127.0.0.1:6829/2749901617] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:54:12.835 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:54:12.836 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 1 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 2 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: 3' 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-10T20:54:12.837 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958531 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958531 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-399431958531' 2024-09-10T20:54:12.838 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285575 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285575 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-399431958530 2-356482285575' 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-10T20:54:13.633 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-399431958530 2-356482285575 3-386547056642' 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: waiting osd.0 seq 21474836536 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836536 -lt 21474836536 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-399431958530 2024-09-10T20:54:13.634 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-399431958530 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958530 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: waiting osd.1 seq 399431958530 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958530 -lt 399431958530 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285575 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:54:13.635 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285575 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285576 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285576 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-399431958531 2-356482285576' 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056643 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056643 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-399431958531 2-356482285576 3-386547056643' 2024-09-10T20:54:15.214 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: waiting osd.0 seq 21474836537 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836536 -lt 21474836537 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:15.215 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836537 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-399431958531 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-399431958531 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958531 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: waiting osd.1 seq 399431958531 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958531 -lt 399431958531 2024-09-10T20:54:15.216 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:16.164 INFO:tasks.workunit.client.0.smithi181.stderr:17744est/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:16.164 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285575 2024-09-10T20:54:16.164 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285575' 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: waiting osd.2 seq 356482285575 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285574 -lt 356482285575 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285576 -lt 356482285575 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056642 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056642 2024-09-10T20:54:16.165 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056642' 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: waiting osd.3 seq 386547056642 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:16.166 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:16.167 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:16.167 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:16.167 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T20:54:16.167 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.8: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285576 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285576 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285576 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285576' 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: waiting osd.2 seq 356482285576 2024-09-10T20:54:16.490 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285576 -lt 356482285576 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056643 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056643 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056643 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056643' 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: waiting osd.3 seq 386547056643 2024-09-10T20:54:16.491 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056643 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:16.492 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:16.493 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T20:54:16.493 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:17.403 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.clclient.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 2 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:17.406 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shient.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:54:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 2 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:54:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:18.900 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:18.901 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:19.192 INFO:tasks.workunit.client.0.smithi181.stderr:177446: //home/ubuntu/cephtest/clone.client.0/qa/sta359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:54:19.193 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:19.194 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:19.195 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:19.251 INFO:tasks.workunit.client.0.smithi181.stderr:177448: //home/ubuntu/cephtest/clone.client.0/qa/standndalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:19.251 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:19.251 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:19.251 INFO:tasks.workunit.client.0.smithi181.stderr:177446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:19.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:19.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:54:19.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 177444 2024-09-10T20:54:19.538 INFO:tasks.workunit.client.0.smithi181.stderr:alone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:177448: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T20:54:19.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T20:54:19.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T20:54:19.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:54:19.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T20:54:19.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T20:54:19.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T20:54:19.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T20:54:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:20.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T20:54:20.367 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T20:54:20.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:20.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T20:54:20.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:20.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T20:54:20.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:20.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:20.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:20.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:54:21.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:54:21.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:21.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:21.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:21.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:21.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:21.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:21.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:21.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:54:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:54:22.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:22.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:22.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:22.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:22.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:22.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:23.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:23.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:54:24.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:54:24.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:24.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:24.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:24.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:24.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:24.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:24.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:24.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:54:25.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:54:25.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:25.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:25.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:25.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:25.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:25.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:25.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:53:43.617067+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:25.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T20:54:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T20:54:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T20:54:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T20:54:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T20:54:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T20:54:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T20:54:26.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T20:54:27.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T20:54:20.830358+0000 '>' 2024-09-10T20:53:43.617067+0000 2024-09-10T20:54:27.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T20:54:27.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 180668"' 2024-09-10T20:54:29.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 180668' 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 180670"' 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 180670' 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 180668 180670' 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 180668 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/180671: /' 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:54:29.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/180673: /' 2024-09-10T20:54:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: _ 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: hinfo_key 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: snapset 2024-09-10T20:54:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:54:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:54:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:180673: start osd.1 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:54:30.233 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: _ 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: hinfo_key 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: snapset 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:54:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:54:30.235 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:54:30.236 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:54:30.237 INFO:tasks.workunit.client.0.smithi181.stderr:180671: start osd.3 2024-09-10T20:54:34.671 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2024-09-10T20:54:30.260+0000 7fd32821b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2024-09-10T20:54:30.263+0000 7fd32821b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2024-09-10T20:54:30.264+0000 7fd32821b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.672 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 0 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:54:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 1 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2024-09-10T20:54:31.892+0000 7fd32821b640 -1 Falling back to public interface 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.674 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:54:34.675 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 3 2024-09-10T20:54:34.675 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.675 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2024-09-10T20:54:34.667+0000 7fd328very_ops 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2024-09-10T20:54:30.209+0000 7f81ab640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2024-09-10T20:54:30.228+0000 7f81ab640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.763 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2024-09-10T20:54:30.230+0000 7f81ab640640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 0 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2024-09-10T20:54:31.591+0000 7f81ab640640 -1 Falling back to public interface 2024-09-10T20:54:34.764 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 1 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2024-09-10T20:54:34.175+0000 7f81ab640640 -1 osd.1 95 log_to_monitors true 2024-09-10T20:54:34.765 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:54:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 3 2024-09-10T20:54:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:34.766 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ce21b640 -1 osd.3 95 log_to_monitors true 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 4 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: osd.3 up in weight 1 up_from 103 up_thru 93 down_at 96 last_clean_interval [90,95) [v2:127.0.0.1:6826/4037405439,v1:127.0.0.1:6827/4037405439] [v2:127.0.0.1:6828/4037405439,v1:127.0.0.1:6829/4037405439] exists,up c55c48f2-4dba-4d29-92ad-cad6f1ab8b78 2024-09-10T20:54:37.130 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:54:37.131 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 1 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 2 2024-09-10T20:54:37.132 INFO:tasks.workunit.client.0.smithi181.stderr:180671: 3' 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-10T20:54:37.133 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 4 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: osd.1 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [93,95) [v2:127.0.0.1:6818/2610848337,v1:127.0.0.1:6819/2610848337] [v2:127.0.0.1:6820/2610848337,v1:127.0.0.1:6821/2610848337] exists,up a452dd8e-8cce-4e2e-9c5c-8eb793f15ffd 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T20:54:37.134 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:54:37.135 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 1 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 2 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: 3' 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836544 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836544 2024-09-10T20:54:37.136 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544' 2024-09-10T20:54:37.137 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.343 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:54:39.343 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729603 2024-09-10T20:54:39.343 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729603 2024-09-10T20:54:39.343 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-429496729603' 2024-09-10T20:54:39.343 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285582 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285582 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-429496729603 2-356482285582' 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631491 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631491 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-429496729603 2-356482285582 3-442381631491' 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836544 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:54:39.344 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836544 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836544 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: waiting osd.0 seq 21474836544 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836544 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836544 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-429496729603 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:39.345 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-429496729603 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:1806.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-429496729602' 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.346 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285583 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285583 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-429496729602 2-356482285583' 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631490 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631490 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-429496729602 2-356482285583 3-442381631490' 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-10T20:54:39.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: waiting osd.0 seq 21474836543 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836543 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-429496729602 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:54:39.348 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-429496729602 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ub73: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729603 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 429496729603' 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: waiting osd.1 seq 429496729603 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729603 -lt 429496729603 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285582 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285582 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285582 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285582' 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: waiting osd.2 seq 356482285582 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285583 -lt 356482285582 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-442381631491 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-442381631491 2024-09-10T20:54:40.662 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631491 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 442381631491' 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: waiting osd.3 seq 442381631491 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631491 -lt 442381631491 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='selectuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 429496729602' 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: waiting osd.1 seq 429496729602 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729603 -lt 429496729602 2024-09-10T20:54:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-356482285583 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-356482285583 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285583 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 356482285583' 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: waiting osd.2 seq 356482285583 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285583 -lt 356482285583 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-442381631490 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:54:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-442381631490 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631490 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 442381631490' 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: waiting osd.3 seq 442381631490 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631491 -lt 442381631490 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T20:54:40.704 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:54:40.705 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:54:40.705 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:54:40.705 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/ce(contains("stale") | not)' 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:41.322 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:41.323 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:41.323 INFO:tasks.workunit.client.0.smithi181.stderr:180673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:41.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:phtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:180671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T20:54:41.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 180670 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T20:54:41.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T20:54:41.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:41.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:41.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:41.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:41.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:41.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:54:41.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:54:41.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:54:41.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:54:41.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:54:41.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:54:41.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:54:41.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:54:41.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:54:41.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:54:41.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:54:41.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:54:41.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:54:41.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:54:41.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:41.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:41.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T20:54:41.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:54:41.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:54:41.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:54:41.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:54:41.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:54:41.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:54:41.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:54:41.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:54:41.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:54:41.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:54:41.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:54:41.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:54:41.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:54:41.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:54:41.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:54:41.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:54:41.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T20:54:41.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:54:41.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:41.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:41.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T20:54:41.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:54:41.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:54:41.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:54:41.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T20:54:41.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T20:54:41.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:41.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:54:42.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:54:42.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:54:42.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:54:42.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:54:42.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:54:42.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:54:42.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T20:54:42.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:54:42.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:54:42.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:54:42.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:54:42.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:54:42.257 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:54:42.257 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:54:42.257 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:54:42.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T20:54:42.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:54:43.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:54:43.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:43.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:54:43.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:54:43.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-10T20:54:43.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:54:43.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-10T20:54:43.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:54:43.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:54:43.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3082e5cd-9c95-4dda-8510-cda2a8fb2477 2024-09-10T20:54:43.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3082e5cd-9c95-4dda-8510-cda2a8fb2477' 2024-09-10T20:54:43.177 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 3082e5cd-9c95-4dda-8510-cda2a8fb2477 2024-09-10T20:54:43.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:54:43.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATsuBmZ81HCxAAXP8QhKNDp7G63zAagdODEQ== 2024-09-10T20:54:43.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATsuBmZ81HCxAAXP8QhKNDp7G63zAagdODEQ=="}' 2024-09-10T20:54:43.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3082e5cd-9c95-4dda-8510-cda2a8fb2477 -i td/osd-scrub-repair/0/new.json 2024-09-10T20:54:43.396 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:54:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T20:54:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATsuBmZ81HCxAAXP8QhKNDp7G63zAagdODEQ== --osd-uuid 3082e5cd-9c95-4dda-8510-cda2a8fb2477 2024-09-10T20:54:43.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:43.432+0000 7fa8c8218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:43.446 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:43.443+0000 7fa8c8218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:43.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:43.444+0000 7fa8c8218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:43.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:43.444+0000 7fa8c8218640 -1 bdev(0x5634756c3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:54:43.448 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:43.444+0000 7fa8c8218640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T20:54:49.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T20:54:49.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:54:49.131 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T20:54:49.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:54:49.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:54:49.494 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T20:54:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:54:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:54:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:54:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:54:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:54:49.522 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:49.518+0000 7f6052a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:49.524 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:49.521+0000 7f6052a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:49.525 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:49.522+0000 7f6052a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:49.773 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:54:49.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:54:49.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:49.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:50.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:51.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:51.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:51.075 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:54:51.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:54:51.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:51.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:51.143 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:51.140+0000 7f6052a0d640 -1 Falling back to public interface 2024-09-10T20:54:51.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:52.334 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:54:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:54:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:52.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:52.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:54:52.788+0000 7f6052a0d640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:54:53.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:53.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:53.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:54:53.619 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:54:53.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:53.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:54.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:55.003 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:54:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:54:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:55.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:55.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:56.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:56.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:56.294 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:54:56.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:54:56.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:56.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:56.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:57.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:57.636 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:54:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:54:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:57.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:57.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:54:58.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:54:58.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:54:58.920 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:54:58.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:54:58.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:54:58.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:54:59.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:00.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:00.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:00.208 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:55:00.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:55:00.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:00.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:00.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:01.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:01.496 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:55:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:55:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:01.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:01.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:02.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:02.799 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:55:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:55:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:03.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:04.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:04.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:04.084 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:55:04.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:55:04.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:04.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:04.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:05.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:05.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:05.624 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:55:05.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:55:05.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:05.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:06.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:07.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:07.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:07.175 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:55:07.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:55:07.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:07.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:07.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:08.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:08.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:08.995 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:55:08.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:55:08.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:08.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:09.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:10.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:10.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:10.522 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:55:10.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:55:10.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:10.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:11.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:12.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:12.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:12.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:55:12.116 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:55:12.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:12.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:13.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:13.426 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:55:13.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:13.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:55:13.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:13.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:15.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:15.311 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:55:15.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:15.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:55:15.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:15.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:55:15.593 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2950816379,v1:127.0.0.1:6803/2950816379] [v2:127.0.0.1:6804/2950816379,v1:127.0.0.1:6805/2950816379] exists,up 3082e5cd-9c95-4dda-8510-cda2a8fb2477 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:55:15.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:55:15.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:55:15.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dca4f197-7533-472a-a049-52e38ee9757e 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dca4f197-7533-472a-a049-52e38ee9757e' 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 dca4f197-7533-472a-a049-52e38ee9757e 2024-09-10T20:55:15.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:55:15.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzsuBmZw1yJBAA8vNz3m54RDRvvMlbpbbwHA== 2024-09-10T20:55:15.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzsuBmZw1yJBAA8vNz3m54RDRvvMlbpbbwHA=="}' 2024-09-10T20:55:15.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dca4f197-7533-472a-a049-52e38ee9757e -i td/osd-scrub-repair/1/new.json 2024-09-10T20:55:15.938 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:55:15.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T20:55:15.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzsuBmZw1yJBAA8vNz3m54RDRvvMlbpbbwHA== --osd-uuid dca4f197-7533-472a-a049-52e38ee9757e 2024-09-10T20:55:15.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:15.977+0000 7f4c10da4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:16.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:16.011+0000 7f4c10da4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:16.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:16.012+0000 7f4c10da4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:16.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:16.012+0000 7f4c10da4640 -1 bdev(0x5581ec865c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:55:16.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:16.012+0000 7f4c10da4640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T20:55:21.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T20:55:21.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:55:21.647 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T20:55:21.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:55:21.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:55:22.008 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T20:55:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:55:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:55:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:55:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:55:22.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:55:22.036 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:22.032+0000 7f9d25b43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:22.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:22.034+0000 7f9d25b43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:22.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:22.035+0000 7f9d25b43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:55:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:55:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:55:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:55:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:55:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:22.287 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:55:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:55:22.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:22.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:22.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:23.407 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:23.403+0000 7f9d25b43640 -1 Falling back to public interface 2024-09-10T20:55:23.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:23.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:23.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:55:23.569 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:55:23.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:23.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:23.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:24.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:24.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:24.853 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:55:24.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:55:24.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:24.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:24.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:24.984+0000 7f9d25b43640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:55:25.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:26.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:26.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:26.207 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:55:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:55:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:26.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:27.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:27.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:27.494 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:55:27.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:55:27.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:27.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:27.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:28.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:28.792 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:55:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:55:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:29.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:30.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:30.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:30.082 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:55:30.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:55:30.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:30.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:30.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:31.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:31.372 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:55:31.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:31.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:55:31.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:31.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:31.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:32.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:32.665 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:55:32.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:32.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:55:32.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:32.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:32.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:33.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:33.974 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:55:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:55:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:33.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:34.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:35.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:35.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:35.286 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:55:35.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:55:35.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:35.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:35.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:36.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:36.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:36.576 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:55:36.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:55:36.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:36.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:36.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:37.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:37.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:37.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:55:37.957 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:55:37.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:37.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:38.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:39.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:39.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:39.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:55:39.556 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:55:39.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:39.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:39.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:40.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:40.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:40.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:55:40.881 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:55:40.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:40.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:41.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:42.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:42.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:42.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:55:42.266 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:55:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:43.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:44.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:44.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:44.022 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:55:44.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:55:44.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:44.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:44.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:45.754 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:55:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:55:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:46.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:47.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:47.393 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:55:47.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:47.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:55:47.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:47.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:47.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:48.679 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:55:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:55:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:55:48.966 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1535163764,v1:127.0.0.1:6811/1535163764] [v2:127.0.0.1:6812/1535163764,v1:127.0.0.1:6813/1535163764] exists,up dca4f197-7533-472a-a049-52e38ee9757e 2024-09-10T20:55:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:55:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:55:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:55:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T20:55:48.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:55:48.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5a2ffb6b-8076-4898-94cb-8d693328d0a5 2024-09-10T20:55:48.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5a2ffb6b-8076-4898-94cb-8d693328d0a5' 2024-09-10T20:55:48.974 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 5a2ffb6b-8076-4898-94cb-8d693328d0a5 2024-09-10T20:55:48.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:55:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUsuBmj9S4OhAA0iwA2FZegf7L9gMVCapRnQ== 2024-09-10T20:55:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUsuBmj9S4OhAA0iwA2FZegf7L9gMVCapRnQ=="}' 2024-09-10T20:55:48.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5a2ffb6b-8076-4898-94cb-8d693328d0a5 -i td/osd-scrub-repair/2/new.json 2024-09-10T20:55:49.377 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:55:49.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T20:55:49.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUsuBmj9S4OhAA0iwA2FZegf7L9gMVCapRnQ== --osd-uuid 5a2ffb6b-8076-4898-94cb-8d693328d0a5 2024-09-10T20:55:49.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:49.413+0000 7f873ad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:49.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:49.448+0000 7f873ad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:49.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:49.449+0000 7f873ad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:49.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:49.450+0000 7f873ad89640 -1 bdev(0x55edc8505c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:55:49.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:49.450+0000 7f873ad89640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T20:55:55.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T20:55:55.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:55:55.153 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T20:55:55.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:55:55.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:55:55.545 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T20:55:55.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:55:55.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:55:55.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:55:55.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:55:55.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:55:55.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:55.569+0000 7f0d2b143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:55.575 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:55.570+0000 7f0d2b143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:55.576 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:55.571+0000 7f0d2b143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:55:55.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:55.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:55:55.822 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:55:55.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:55.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:55:56.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:56.923 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:56.919+0000 7f0d2b143640 -1 Falling back to public interface 2024-09-10T20:55:57.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:57.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:57.097 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:55:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:55:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:57.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:55:57.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:58.409 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:55:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:55:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:58.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:55:58.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:55:58.503+0000 7f0d2b143640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:55:58.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:55:59.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:55:59.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:55:59.717 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:55:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:55:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:55:59.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:01.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:01.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:01.003 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:56:01.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:56:01.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:01.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:01.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:02.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:02.412 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:56:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:56:02.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:02.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:02.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:03.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:03.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:03.701 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:56:03.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:56:03.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:03.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:03.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:04.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:04.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:04.988 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:56:04.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:56:04.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:04.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:05.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:06.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:06.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:06.430 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:56:06.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:56:06.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:06.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:06.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:07.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:07.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:07.717 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:56:07.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:56:07.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:07.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:08.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:09.030 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:56:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:09.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:56:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:09.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:09.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:10.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:10.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:10.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:56:10.390 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:56:10.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:10.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:11.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:11.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:11.674 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:56:11.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:56:11.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:11.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:12.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:13.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:13.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:13.461 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:56:13.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:56:13.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:13.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:14.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:15.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:15.355 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:56:15.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:15.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:56:15.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:15.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:16.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:17.215 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:56:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:56:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:18.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:19.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:19.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:19.009 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:56:19.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:56:19.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:19.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:19.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:20.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:20.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:20.606 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:56:20.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:56:20.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:20.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:21.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:22.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:22.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:22.257 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:56:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:56:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:56:22.541 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3904106414,v1:127.0.0.1:6819/3904106414] [v2:127.0.0.1:6820/3904106414,v1:127.0.0.1:6821/3904106414] exists,up 5a2ffb6b-8076-4898-94cb-8d693328d0a5 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T20:56:22.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:56:22.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:56:22.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T20:56:22.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a0e4d625-7d36-4c91-9851-31e1f8a00a28' 2024-09-10T20:56:22.549 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T20:56:22.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:56:22.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2suBmI8pnIRAAsJgfrBxHQH5kMbfmRKgybQ== 2024-09-10T20:56:22.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2suBmI8pnIRAAsJgfrBxHQH5kMbfmRKgybQ=="}' 2024-09-10T20:56:22.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0e4d625-7d36-4c91-9851-31e1f8a00a28 -i td/osd-scrub-repair/3/new.json 2024-09-10T20:56:23.020 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:56:23.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T20:56:23.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2suBmI8pnIRAAsJgfrBxHQH5kMbfmRKgybQ== --osd-uuid a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T20:56:23.062 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:23.058+0000 7f0d2f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:23.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:23.118+0000 7f0d2f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:23.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:23.119+0000 7f0d2f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:23.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:23.119+0000 7f0d2f011640 -1 bdev(0x562df7409c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:56:23.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:23.119+0000 7f0d2f011640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T20:56:29.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T20:56:29.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:56:29.834 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T20:56:29.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:56:29.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:56:30.198 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T20:56:30.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:56:30.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:56:30.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:56:30.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:56:30.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:56:30.229 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:30.222+0000 7f3709818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:30.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:30.224+0000 7f3709818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:30.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:30.226+0000 7f3709818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:30.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:56:30.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:56:30.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:30.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:31.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:31.750 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:56:31.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:56:31.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:31.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:32.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:32.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:32.090+0000 7f3709818640 -1 Falling back to public interface 2024-09-10T20:56:33.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:33.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:33.034 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:56:33.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:56:33.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:33.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:33.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:33.871 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:33.868+0000 7f3709818640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:56:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:34.330 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:56:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:56:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:34.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:35.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:35.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:35.675 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:56:35.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:56:35.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:35.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:36.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:37.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:37.087 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:56:37.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:37.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:56:37.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:37.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:37.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:38.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:38.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:38.408 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:56:38.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:56:38.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:38.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:38.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:39.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:39.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:39.716 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:56:39.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:56:39.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:39.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:40.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:41.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:41.003 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:56:41.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:41.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:56:41.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:41.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:41.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:42.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:42.292 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:56:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:56:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:42.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:42.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:43.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:43.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:43.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:56:43.687 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:56:43.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:43.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:44.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:45.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:45.014 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:56:45.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:45.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:56:45.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:45.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:45.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:46.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:46.306 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:56:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:56:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:46.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:47.650 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:56:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:56:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:48.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:56:49.504 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:49.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:50.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:51.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:51.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:51.251 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:56:51.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:56:51.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:51.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:52.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:53.118 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:56:53.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:53.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:56:53.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:53.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:53.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:54.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:54.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:54.524 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:56:54.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:56:54.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:54.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:55.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:56.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:56.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:56.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:56:56.189 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:56:56.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:56.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:56.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:56:57.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:56:57.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:56:57.827 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:56:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:56:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:56:57.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:56:58.111 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2564377640,v1:127.0.0.1:6827/2564377640] [v2:127.0.0.1:6828/2564377640,v1:127.0.0.1:6829/2564377640] exists,up a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T20:56:58.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:56:58.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:56:58.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:56:58.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:56:58.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:56:58.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:56:58.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-10T20:56:58.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:56:58.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12b02da0-f9fb-4417-bc5d-ff5faa99d717 2024-09-10T20:56:58.119 INFO:tasks.workunit.client.0.smithi181.stdout:add osd4 12b02da0-f9fb-4417-bc5d-ff5faa99d717 2024-09-10T20:56:58.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 12b02da0-f9fb-4417-bc5d-ff5faa99d717' 2024-09-10T20:56:58.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:56:58.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCasuBmI+O9BxAAjOXqDFhQMr/WcWie6k/D/A== 2024-09-10T20:56:58.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCasuBmI+O9BxAAjOXqDFhQMr/WcWie6k/D/A=="}' 2024-09-10T20:56:58.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12b02da0-f9fb-4417-bc5d-ff5faa99d717 -i td/osd-scrub-repair/4/new.json 2024-09-10T20:56:58.485 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:56:58.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-10T20:56:58.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCasuBmI+O9BxAAjOXqDFhQMr/WcWie6k/D/A== --osd-uuid 12b02da0-f9fb-4417-bc5d-ff5faa99d717 2024-09-10T20:56:58.527 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:58.523+0000 7f148c727640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:58.574+0000 7f148c727640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:58.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:58.575+0000 7f148c727640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:56:58.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:58.576+0000 7f148c727640 -1 bdev(0x5581892e1c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-10T20:56:58.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:56:58.576+0000 7f148c727640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-10T20:57:04.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-10T20:57:04.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:57:04.603 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd4 key to auth repository 2024-09-10T20:57:04.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T20:57:04.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:57:04.962 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.4 2024-09-10T20:57:04.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T20:57:04.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:57:04.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:57:04.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:57:04.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:57:04.991 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:04.987+0000 7fbae1745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:04.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:04.989+0000 7fbae1745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:04.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:04.990+0000 7fbae1745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:05.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:05.241 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:57:05.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:57:05.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:05.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:05.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:06.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:06.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:06.535 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:57:06.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:57:06.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:06.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:06.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:06.588+0000 7fbae1745640 -1 Falling back to public interface 2024-09-10T20:57:06.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:07.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:07.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:07.822 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:57:07.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:57:07.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:07.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:08.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:08.184 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:08.180+0000 7fbae1745640 -1 osd.4 0 log_to_monitors true 2024-09-10T20:57:09.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:09.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:09.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:57:09.109 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:57:09.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:09.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:09.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:10.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:10.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:10.417 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:57:10.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:57:10.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:10.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:10.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:11.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:11.705 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:57:11.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:57:11.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:11.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:11.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:12.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:12.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:12.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:57:12.992 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:57:12.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:12.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:13.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:14.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:14.292 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:57:14.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:57:14.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:14.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:14.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:15.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:15.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:15.581 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:57:15.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:57:15.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:15.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:15.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:16.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:16.885 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:57:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:57:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:17.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:18.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:18.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:18.275 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:57:18.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:57:18.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:18.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:18.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:19.564 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:57:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:57:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:19.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:20.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:20.888 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:57:20.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:20.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:57:20.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:20.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:21.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:22.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:22.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:22.205 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:57:22.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:57:22.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:22.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:22.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:23.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:23.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:23.677 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:57:23.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:57:23.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:23.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:24.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:57:25.615 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:57:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:26.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:27.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:27.401 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:57:27.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:27.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:57:27.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:27.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:28.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:29.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:29.171 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:57:29.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:57:29.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:29.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:29.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:30.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:30.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:30.557 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:57:30.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:57:30.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:30.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:31.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:32.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:32.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:32.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:57:32.303 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:57:32.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:32.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:57:32.602 INFO:tasks.workunit.client.0.smithi181.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2550417404,v1:127.0.0.1:6835/2550417404] [v2:127.0.0.1:6836/2550417404,v1:127.0.0.1:6837/2550417404] exists,up 12b02da0-f9fb-4417-bc5d-ff5faa99d717 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:57:32.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:57:32.607 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:57:32.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92' 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stdout:add osd5 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T20:57:32.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:57:32.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC8suBmFqULJRAA90VrR7xvV2TP9VxeF5RnBw== 2024-09-10T20:57:32.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC8suBmFqULJRAA90VrR7xvV2TP9VxeF5RnBw=="}' 2024-09-10T20:57:32.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 -i td/osd-scrub-repair/5/new.json 2024-09-10T20:57:33.088 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:57:33.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-10T20:57:33.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8suBmFqULJRAA90VrR7xvV2TP9VxeF5RnBw== --osd-uuid 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T20:57:33.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:33.125+0000 7f6247bdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:33.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:33.193+0000 7f6247bdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:33.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:33.195+0000 7f6247bdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:33.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:33.195+0000 7f6247bdd640 -1 bdev(0x561f87557c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-10T20:57:33.200 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:33.195+0000 7f6247bdd640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-10T20:57:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-10T20:57:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:57:39.110 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd5 key to auth repository 2024-09-10T20:57:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T20:57:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:57:39.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T20:57:39.483 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.5 2024-09-10T20:57:39.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:57:39.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:57:39.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:57:39.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:57:39.511 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:39.507+0000 7f751be31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:39.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:39.509+0000 7f751be31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:39.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:39.510+0000 7f751be31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:57:39.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:57:39.762 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:57:39.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:39.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:40.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:41.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:41.037 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:57:41.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:41.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:57:41.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:41.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:41.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:41.364 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:41.361+0000 7f751be31640 -1 Falling back to public interface 2024-09-10T20:57:42.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:42.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:42.327 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:57:42.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:57:42.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:42.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:42.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:43.263 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:57:43.260+0000 7f751be31640 -1 osd.5 0 log_to_monitors true 2024-09-10T20:57:43.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:43.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:43.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:57:43.615 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:57:43.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:43.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:43.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:44.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:44.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:44.995 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:57:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:57:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:45.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:46.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:46.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:46.326 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:57:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:57:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:46.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:46.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:47.619 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:57:47.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:57:47.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:47.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:48.907 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:57:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:57:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:49.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:50.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:50.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:50.311 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:57:50.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:57:50.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:50.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:50.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:51.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:51.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:51.603 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:57:51.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:57:51.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:51.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:51.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:52.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:52.932 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:57:52.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:52.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:57:52.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:52.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:53.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:54.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:54.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:54.296 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:57:54.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:57:54.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:54.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:54.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:55.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:55.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:55.588 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:57:55.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:57:55.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:55.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:56.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:57.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:57.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:57.048 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:57:57.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:57:57.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:57.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:57.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:57:58.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:57:58.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:57:58.417 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:57:58.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:57:58.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:57:58.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:57:59.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:00.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:00.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:00.145 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:58:00.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:58:00.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:00.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:58:00.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:01.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:01.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:01.447 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:58:01.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:58:01.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:01.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:58:02.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:03.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:03.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:03.017 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:58:03.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:58:03.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:03.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:58:03.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:04.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:04.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:04.416 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:58:04.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:58:04.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:04.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:58:05.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:06.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:06.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:06.155 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:58:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:58:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:58:06.505 INFO:tasks.workunit.client.0.smithi181.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3062710658,v1:127.0.0.1:6843/3062710658] [v2:127.0.0.1:6844/3062710658,v1:127.0.0.1:6845/3062710658] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T20:58:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:58:06.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:58:06.511 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:58:06.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=063f7b19-8d94-4158-8735-696ad0bed60a 2024-09-10T20:58:06.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 063f7b19-8d94-4158-8735-696ad0bed60a' 2024-09-10T20:58:06.514 INFO:tasks.workunit.client.0.smithi181.stdout:add osd6 063f7b19-8d94-4158-8735-696ad0bed60a 2024-09-10T20:58:06.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:58:06.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDesuBmsyc0HxAAnXAeUR54go2z7l7kkm59mA== 2024-09-10T20:58:06.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDesuBmsyc0HxAAnXAeUR54go2z7l7kkm59mA=="}' 2024-09-10T20:58:06.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 063f7b19-8d94-4158-8735-696ad0bed60a -i td/osd-scrub-repair/6/new.json 2024-09-10T20:58:06.857 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:58:06.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-10T20:58:06.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDesuBmsyc0HxAAnXAeUR54go2z7l7kkm59mA== --osd-uuid 063f7b19-8d94-4158-8735-696ad0bed60a 2024-09-10T20:58:06.899 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:06.895+0000 7fb07a789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:06.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:06.955+0000 7fb07a789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:06.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:06.956+0000 7fb07a789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:06.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:06.956+0000 7fb07a789640 -1 bdev(0x56107c1fdc00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-10T20:58:06.960 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:06.956+0000 7fb07a789640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-10T20:58:13.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-10T20:58:13.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:58:13.358 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd6 key to auth repository 2024-09-10T20:58:13.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T20:58:13.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:58:13.723 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.6 2024-09-10T20:58:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T20:58:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:58:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:58:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:58:13.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:58:13.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:13.747+0000 7f3afae85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:13.752 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:13.749+0000 7f3afae85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:13.750+0000 7f3afae85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:14.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:58:14.001 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:58:14.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:14.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:14.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:15.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:15.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:15.276 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:58:15.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:58:15.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:15.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:15.602+0000 7f3afae85640 -1 Falling back to public interface 2024-09-10T20:58:16.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:16.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:16.567 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:58:16.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:58:16.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:16.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:16.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:17.631 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:17.627+0000 7f3afae85640 -1 osd.6 0 log_to_monitors true 2024-09-10T20:58:17.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:17.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:17.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:58:17.865 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:58:17.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:17.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:18.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:19.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:19.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:19.191 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:58:19.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:58:19.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:19.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:20.479 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:58:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:58:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:20.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:21.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:21.767 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:58:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:58:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:22.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:23.054 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:58:23.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:58:23.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:23.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:23.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:24.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:24.386 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:58:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:58:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:24.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:25.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:25.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:25.673 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:58:25.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:58:25.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:25.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:25.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:26.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:58:26.964 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:58:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:26.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:27.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:28.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:28.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:28.261 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:58:28.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:58:28.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:28.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:28.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:29.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:29.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:29.556 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:58:29.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:58:29.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:29.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:29.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:30.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:30.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:30.842 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:58:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:58:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:31.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:32.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:32.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:32.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:58:32.132 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:58:32.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:32.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:32.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:33.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:33.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:33.930 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:58:33.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:58:33.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:33.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:34.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:35.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:35.743 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:58:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:58:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:35.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:36.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:37.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:37.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:37.350 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:58:37.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:58:37.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:37.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:37.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:38.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:38.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:38.745 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:58:38.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:58:38.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:38.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:39.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:40.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:40.152 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:58:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:40.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:58:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:40.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T20:58:40.480 INFO:tasks.workunit.client.0.smithi181.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/300750527,v1:127.0.0.1:6851/300750527] [v2:127.0.0.1:6852/300750527,v1:127.0.0.1:6853/300750527] exists,up 063f7b19-8d94-4158-8735-696ad0bed60a 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:58:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:58:40.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:58:40.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4be00b96-ab87-4afd-bca6-1d05d2c92e09 2024-09-10T20:58:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 4be00b96-ab87-4afd-bca6-1d05d2c92e09' 2024-09-10T20:58:40.488 INFO:tasks.workunit.client.0.smithi181.stdout:add osd7 4be00b96-ab87-4afd-bca6-1d05d2c92e09 2024-09-10T20:58:40.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:58:40.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAs+Bm/P+8HRAAT0TxYh7aP1f8wF4IoMTSwg== 2024-09-10T20:58:40.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAs+Bm/P+8HRAAT0TxYh7aP1f8wF4IoMTSwg=="}' 2024-09-10T20:58:40.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4be00b96-ab87-4afd-bca6-1d05d2c92e09 -i td/osd-scrub-repair/7/new.json 2024-09-10T20:58:40.919 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:58:40.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-10T20:58:40.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAs+Bm/P+8HRAAT0TxYh7aP1f8wF4IoMTSwg== --osd-uuid 4be00b96-ab87-4afd-bca6-1d05d2c92e09 2024-09-10T20:58:40.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:40.957+0000 7f8ff8819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:40.970 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:40.967+0000 7f8ff8819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:40.971 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:40.968+0000 7f8ff8819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:40.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:40.968+0000 7f8ff8819640 -1 bdev(0x55940b4dfc00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-10T20:58:40.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:40.968+0000 7f8ff8819640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-10T20:58:47.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-10T20:58:47.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:58:47.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-10T20:58:47.145 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd7 key to auth repository 2024-09-10T20:58:47.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:58:47.509 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.7 2024-09-10T20:58:47.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-10T20:58:47.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:58:47.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:58:47.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:58:47.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:58:47.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:47.534+0000 7ff11fcf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:47.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:47.535+0000 7ff11fcf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:47.540 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:47.537+0000 7ff11fcf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:47.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:58:47.787 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:58:47.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:47.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:48.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:48.898 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:48.894+0000 7ff11fcf1640 -1 Falling back to public interface 2024-09-10T20:58:49.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:49.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:49.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:58:49.065 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:58:49.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:49.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:49.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:50.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:50.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:50.370 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:58:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:58:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:50.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:50.498 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:58:50.494+0000 7ff11fcf1640 -1 osd.7 0 log_to_monitors true 2024-09-10T20:58:50.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:51.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:51.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:51.708 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:58:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:58:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:51.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:52.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:52.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:52.997 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:58:52.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:58:52.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:52.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:53.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:54.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:54.314 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:58:54.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:58:54.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:54.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:54.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:55.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:55.594 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:58:55.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:58:55.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:55.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:55.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:56.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:56.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:56.884 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:58:56.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:58:56.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:56.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:57.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:58.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:58.174 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:58:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:58:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:58.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:58.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:58:59.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:58:59.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:58:59.525 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:58:59.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:58:59.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:58:59.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:58:59.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:00.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:00.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:00.930 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:59:00.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:59:00.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:00.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:01.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:02.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:02.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:02.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:59:02.219 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:59:02.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:02.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:02.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:03.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:03.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:03.524 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:59:03.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:59:03.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:03.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:05.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:05.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:05.455 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:59:05.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:59:05.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:05.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:05.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:06.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:06.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:06.934 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:59:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:59:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:07.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:08.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:08.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:08.404 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:59:08.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:59:08.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:08.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:08.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:09.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:09.943 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:59:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:59:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:09.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:10.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:11.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:11.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:11.231 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:59:11.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:59:11.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:11.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:12.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:12.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:12.703 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:59:12.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:59:12.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:12.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:13.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:14.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:14.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:14.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T20:59:14.375 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T20:59:14.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:14.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T20:59:14.669 INFO:tasks.workunit.client.0.smithi181.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3464738250,v1:127.0.0.1:6859/3464738250] [v2:127.0.0.1:6860/3464738250,v1:127.0.0.1:6861/3464738250] exists,up 4be00b96-ab87-4afd-bca6-1d05d2c92e09 2024-09-10T20:59:14.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:59:14.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-10T20:59:14.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:59:14.676 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:59:14.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:59:14.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=62ca95e9-769e-4f53-b767-213cd7dff621 2024-09-10T20:59:14.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 62ca95e9-769e-4f53-b767-213cd7dff621' 2024-09-10T20:59:14.679 INFO:tasks.workunit.client.0.smithi181.stdout:add osd8 62ca95e9-769e-4f53-b767-213cd7dff621 2024-09-10T20:59:14.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:59:14.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAis+Bma2D/KBAAfGBtQPUlRftyQnxRSfTXTQ== 2024-09-10T20:59:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAis+Bma2D/KBAAfGBtQPUlRftyQnxRSfTXTQ=="}' 2024-09-10T20:59:14.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 62ca95e9-769e-4f53-b767-213cd7dff621 -i td/osd-scrub-repair/8/new.json 2024-09-10T20:59:15.040 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:59:15.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-10T20:59:15.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAis+Bma2D/KBAAfGBtQPUlRftyQnxRSfTXTQ== --osd-uuid 62ca95e9-769e-4f53-b767-213cd7dff621 2024-09-10T20:59:15.083 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:15.078+0000 7ff5c12ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:15.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:15.128+0000 7ff5c12ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:15.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:15.129+0000 7ff5c12ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:15.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:15.130+0000 7ff5c12ae640 -1 bdev(0x5601891dfc00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-10T20:59:15.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:15.130+0000 7ff5c12ae640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-10T20:59:22.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-10T20:59:22.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:59:22.207 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd8 key to auth repository 2024-09-10T20:59:22.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-10T20:59:22.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:59:22.568 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.8 2024-09-10T20:59:22.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-10T20:59:22.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:59:22.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:59:22.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:59:22.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:59:22.596 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:22.592+0000 7f482a815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:22.597 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:22.594+0000 7f482a815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:22.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:22.595+0000 7f482a815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:22.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:59:22.845 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:59:22.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:22.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:23.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:23.972 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:23.968+0000 7f482a815640 -1 Falling back to public interface 2024-09-10T20:59:24.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:24.121 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:59:24.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:24.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:59:24.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:24.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:24.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:25.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:25.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:25.402 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:59:25.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:59:25.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:25.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:25.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:25.677+0000 7f482a815640 -1 osd.8 0 log_to_monitors true 2024-09-10T20:59:25.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:26.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:26.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:26.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:59:26.692 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:59:26.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:26.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:26.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:27.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:27.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:27.982 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T20:59:27.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:59:27.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:27.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:28.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:29.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:29.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:29.378 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T20:59:29.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:59:29.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:29.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:29.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:30.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:30.666 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T20:59:30.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:30.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:59:30.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:30.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:30.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:31.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:31.957 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T20:59:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:59:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:32.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:33.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:33.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:33.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:59:33.249 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T20:59:33.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:33.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:33.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:34.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:34.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:34.622 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:59:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:59:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:34.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:34.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:35.910 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T20:59:35.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:35.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:59:35.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:35.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:36.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:37.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:37.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:37.202 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T20:59:37.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:59:37.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:37.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:37.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:38.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:38.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:38.770 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T20:59:38.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:59:38.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:38.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:39.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:40.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:40.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:40.209 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T20:59:40.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:59:40.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:40.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:40.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:41.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:41.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:41.657 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T20:59:41.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:59:41.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:41.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:42.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:43.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:43.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:43.461 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T20:59:43.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:59:43.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:43.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:43.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:44.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:44.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:44.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:59:44.905 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T20:59:44.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:44.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:45.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:46.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:46.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:46.558 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T20:59:46.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:59:46.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:46.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:47.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:48.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:48.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:48.289 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T20:59:48.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:59:48.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:48.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T20:59:48.674 INFO:tasks.workunit.client.0.smithi181.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/89053430,v1:127.0.0.1:6867/89053430] [v2:127.0.0.1:6868/89053430,v1:127.0.0.1:6869/89053430] exists,up 62ca95e9-769e-4f53-b767-213cd7dff621 2024-09-10T20:59:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:59:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:59:48.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:59:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T20:59:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-10T20:59:48.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:59:48.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T20:59:48.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T20:59:48.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:59:48.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T20:59:48.684 INFO:tasks.workunit.client.0.smithi181.stdout:add osd9 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T20:59:48.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 6ac4d393-7386-4fbb-a034-1696d4eeda08' 2024-09-10T20:59:48.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:59:48.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEs+Bme5BQKRAAObkMMN//bNL9iXxu/pBrRg== 2024-09-10T20:59:48.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEs+Bme5BQKRAAObkMMN//bNL9iXxu/pBrRg=="}' 2024-09-10T20:59:48.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ac4d393-7386-4fbb-a034-1696d4eeda08 -i td/osd-scrub-repair/9/new.json 2024-09-10T20:59:49.065 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T20:59:49.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-10T20:59:49.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEs+Bme5BQKRAAObkMMN//bNL9iXxu/pBrRg== --osd-uuid 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T20:59:49.107 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:49.103+0000 7f16f5902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:49.153 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:49.150+0000 7f16f5902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:49.154 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:49.151+0000 7f16f5902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:49.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:49.151+0000 7f16f5902640 -1 bdev(0x55b39518dc00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-10T20:59:49.155 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:49.151+0000 7f16f5902640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-10T20:59:55.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-10T20:59:55.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:59:55.315 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd9 key to auth repository 2024-09-10T20:59:55.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-10T20:59:55.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:59:55.679 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.9 2024-09-10T20:59:55.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-10T20:59:55.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:59:55.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:59:55.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:59:55.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:59:55.706 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:55.702+0000 7fbf9c2d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:55.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:55.705+0000 7fbf9c2d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:55.709 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:55.706+0000 7fbf9c2d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:59:55.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-10T20:59:55.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:59:55.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T20:59:55.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:55.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T20:59:56.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:56.828 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:56.824+0000 7fbf9c2d6640 -1 Falling back to public interface 2024-09-10T20:59:57.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:57.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:57.234 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T20:59:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:59:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:57.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T20:59:57.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:58.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:58.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:58.534 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T20:59:58.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:59:58.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:58.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T20:59:58.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:59:59.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T20:59:59.108+0000 7fbf9c2d6640 -1 osd.9 0 log_to_monitors true 2024-09-10T20:59:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:59:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:59:59.823 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T20:59:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:59:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:59:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:00.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:01.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:01.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:01.204 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:00:01.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:00:01.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:01.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:01.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:00:02.507 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:00:02.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:02.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:03.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:03.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:03.797 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:00:03.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:00:03.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:03.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:04.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:05.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:05.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:05.123 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:00:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:00:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:05.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:06.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:06.424 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:00:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:00:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:06.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:07.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:07.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:07.715 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:00:07.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:00:07.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:07.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:08.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:09.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:09.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:09.121 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:00:09.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:00:09.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:09.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:09.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:10.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:10.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:10.412 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:00:10.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:00:10.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:10.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:10.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:11.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:11.701 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:00:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:00:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:11.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:12.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:13.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:13.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:13.024 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:00:13.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:00:13.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:13.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:14.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:14.832 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:00:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:00:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:14.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:15.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:16.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:16.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:16.326 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:00:16.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:00:16.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:16.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:16.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:17.752 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:00:17.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:00:17.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:17.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:18.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:19.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:19.580 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:00:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:00:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:20.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:21.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:21.418 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:00:21.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:00:21.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:21.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:22.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:23.263 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:00:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:00:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:23.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:24.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:24.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:24.611 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T21:00:24.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T21:00:24.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:24.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:00:24.901 INFO:tasks.workunit.client.0.smithi181.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/478592364,v1:127.0.0.1:6875/478592364] [v2:127.0.0.1:6876/478592364,v1:127.0.0.1:6877/478592364] exists,up 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T21:00:24.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:00:24.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:00:24.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:00:24.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-10T21:00:24.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:00:25.168 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:00:25.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:00:25.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:00:25.557 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:00:25.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:00:26.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:00:29.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:00:29.886 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:00:29.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:00:29.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:00:29.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:00:29.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:00:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:00:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:00:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:00:29.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:00:29.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:00:30.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:00:30.263 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:30.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:00:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836556 2024-09-10T21:00:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836556 2024-09-10T21:00:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556' 2024-09-10T21:00:30.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:30.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:00:30.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-10T21:00:30.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-10T21:00:30.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023' 2024-09-10T21:00:30.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:30.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:00:30.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509496 2024-09-10T21:00:30.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509496 2024-09-10T21:00:30.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496' 2024-09-10T21:00:30.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:30.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:00:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345971 2024-09-10T21:00:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345971 2024-09-10T21:00:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971' 2024-09-10T21:00:30.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:30.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:00:31.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182439 2024-09-10T21:00:31.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182439 2024-09-10T21:00:31.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439' 2024-09-10T21:00:31.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:31.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:00:31.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018912 2024-09-10T21:00:31.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018912 2024-09-10T21:00:31.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439 5-128849018912' 2024-09-10T21:00:31.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:31.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:00:31.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855388 2024-09-10T21:00:31.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855388 2024-09-10T21:00:31.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439 5-128849018912 6-150323855388' 2024-09-10T21:00:31.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:31.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:00:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691864 2024-09-10T21:00:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691864 2024-09-10T21:00:31.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439 5-128849018912 6-150323855388 7-171798691864' 2024-09-10T21:00:31.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:31.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:00:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528333 2024-09-10T21:00:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528333 2024-09-10T21:00:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439 5-128849018912 6-150323855388 7-171798691864 8-193273528333' 2024-09-10T21:00:31.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:31.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:00:31.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364805 2024-09-10T21:00:31.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364805 2024-09-10T21:00:31.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673023 2-64424509496 3-85899345971 4-107374182439 5-128849018912 6-150323855388 7-171798691864 8-193273528333 9-214748364805' 2024-09-10T21:00:31.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:31.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836556 2024-09-10T21:00:31.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:31.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:00:31.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836556 2024-09-10T21:00:31.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:31.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836556 2024-09-10T21:00:31.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-10T21:00:31.523 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836556 2024-09-10T21:00:31.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:00:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-10T21:00:31.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:31.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-10T21:00:31.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:00:31.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-10T21:00:31.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:31.800 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673023 2024-09-10T21:00:31.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-10T21:00:31.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-10T21:00:31.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:00:32.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-10T21:00:32.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:32.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509496 2024-09-10T21:00:32.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:32.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:00:32.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509496 2024-09-10T21:00:32.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:32.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509496 2024-09-10T21:00:32.080 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509496 2024-09-10T21:00:32.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-09-10T21:00:32.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:00:32.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509496 -lt 64424509496 2024-09-10T21:00:32.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:32.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345971 2024-09-10T21:00:32.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:32.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:00:32.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345971 2024-09-10T21:00:32.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:32.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345971 2024-09-10T21:00:32.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-09-10T21:00:32.365 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345971 2024-09-10T21:00:32.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:00:32.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345971 -lt 85899345971 2024-09-10T21:00:32.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:32.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182439 2024-09-10T21:00:32.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:00:32.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182439 2024-09-10T21:00:32.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182439 2024-09-10T21:00:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-09-10T21:00:32.687 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.4 seq 107374182439 2024-09-10T21:00:32.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:00:32.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182439 -lt 107374182439 2024-09-10T21:00:32.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:32.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018912 2024-09-10T21:00:32.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:32.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:00:32.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018912 2024-09-10T21:00:32.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018912 2024-09-10T21:00:32.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018912' 2024-09-10T21:00:32.969 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.5 seq 128849018912 2024-09-10T21:00:32.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:00:33.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018912 -lt 128849018912 2024-09-10T21:00:33.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:33.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855388 2024-09-10T21:00:33.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:33.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:00:33.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855388 2024-09-10T21:00:33.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:33.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855388 2024-09-10T21:00:33.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-09-10T21:00:33.256 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.6 seq 150323855388 2024-09-10T21:00:33.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:00:33.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855388 -lt 150323855388 2024-09-10T21:00:33.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:33.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691864 2024-09-10T21:00:33.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:33.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:00:33.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691864 2024-09-10T21:00:33.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:33.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691864 2024-09-10T21:00:33.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-09-10T21:00:33.536 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.7 seq 171798691864 2024-09-10T21:00:33.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:00:33.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691864 -lt 171798691864 2024-09-10T21:00:33.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:33.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528333 2024-09-10T21:00:33.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:33.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:00:33.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528333 2024-09-10T21:00:33.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:33.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528333 2024-09-10T21:00:33.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-09-10T21:00:33.814 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.8 seq 193273528333 2024-09-10T21:00:33.814 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:00:34.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528333 -lt 193273528333 2024-09-10T21:00:34.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:34.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364805 2024-09-10T21:00:34.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:34.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:00:34.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364805 2024-09-10T21:00:34.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:34.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364805 2024-09-10T21:00:34.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-09-10T21:00:34.096 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.9 seq 214748364805 2024-09-10T21:00:34.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:00:34.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364805 -lt 214748364805 2024-09-10T21:00:34.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:00:34.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:34.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:00:34.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:00:34.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:00:34.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:00:34.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:34.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:00:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-10T21:00:35.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:00:35.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:00:36.438 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:00:36.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:00:37.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:00:37.454 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:00:37.454 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:00:37.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:00:37.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:00:37.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:00:37.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:00:37.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:00:37.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:00:37.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:00:37.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:00:37.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:00:37.840 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:00:37.841 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:00:37.841 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:00:37.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:00:37.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:37.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:00:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836558 2024-09-10T21:00:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836558 2024-09-10T21:00:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558' 2024-09-10T21:00:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:37.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:00:38.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673025 2024-09-10T21:00:38.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673025 2024-09-10T21:00:38.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025' 2024-09-10T21:00:38.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:00:38.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509499 2024-09-10T21:00:38.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509499 2024-09-10T21:00:38.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499' 2024-09-10T21:00:38.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:00:38.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345974 2024-09-10T21:00:38.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345974 2024-09-10T21:00:38.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974' 2024-09-10T21:00:38.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:00:38.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182442 2024-09-10T21:00:38.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182442 2024-09-10T21:00:38.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442' 2024-09-10T21:00:38.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:00:38.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018915 2024-09-10T21:00:38.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018915 2024-09-10T21:00:38.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442 5-128849018915' 2024-09-10T21:00:38.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:00:38.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855390 2024-09-10T21:00:38.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855390 2024-09-10T21:00:38.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442 5-128849018915 6-150323855390' 2024-09-10T21:00:38.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:00:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691866 2024-09-10T21:00:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691866 2024-09-10T21:00:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442 5-128849018915 6-150323855390 7-171798691866' 2024-09-10T21:00:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:38.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:00:39.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528335 2024-09-10T21:00:39.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528335 2024-09-10T21:00:39.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442 5-128849018915 6-150323855390 7-171798691866 8-193273528335' 2024-09-10T21:00:39.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:39.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:00:39.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364808 2024-09-10T21:00:39.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364808 2024-09-10T21:00:39.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673025 2-64424509499 3-85899345974 4-107374182442 5-128849018915 6-150323855390 7-171798691866 8-193273528335 9-214748364808' 2024-09-10T21:00:39.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:39.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836558 2024-09-10T21:00:39.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:39.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:00:39.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836558 2024-09-10T21:00:39.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:39.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836558 2024-09-10T21:00:39.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-10T21:00:39.132 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836558 2024-09-10T21:00:39.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:00:39.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-10T21:00:39.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:39.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673025 2024-09-10T21:00:39.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:39.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:00:39.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673025 2024-09-10T21:00:39.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673025 2024-09-10T21:00:39.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-09-10T21:00:39.416 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673025 2024-09-10T21:00:39.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:00:39.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673025 2024-09-10T21:00:39.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:39.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509499 2024-09-10T21:00:39.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:39.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:00:39.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509499 2024-09-10T21:00:39.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:39.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509499 2024-09-10T21:00:39.722 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509499 2024-09-10T21:00:39.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-09-10T21:00:39.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:00:40.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509499 -lt 64424509499 2024-09-10T21:00:40.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:40.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345974 2024-09-10T21:00:40.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:40.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:00:40.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345974 2024-09-10T21:00:40.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:40.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345974 2024-09-10T21:00:40.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2024-09-10T21:00:40.005 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345974 2024-09-10T21:00:40.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:00:40.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345974 -lt 85899345974 2024-09-10T21:00:40.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:40.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182442 2024-09-10T21:00:40.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:40.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:00:40.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182442 2024-09-10T21:00:40.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:40.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182442 2024-09-10T21:00:40.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-09-10T21:00:40.285 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.4 seq 107374182442 2024-09-10T21:00:40.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:00:40.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182442 -lt 107374182442 2024-09-10T21:00:40.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:40.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018915 2024-09-10T21:00:40.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:40.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:00:40.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018915 2024-09-10T21:00:40.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:40.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018915 2024-09-10T21:00:40.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-09-10T21:00:40.602 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.5 seq 128849018915 2024-09-10T21:00:40.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018915 -lt 128849018915 2024-09-10T21:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855390 2024-09-10T21:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:40.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:00:40.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855390 2024-09-10T21:00:40.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:40.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855390 2024-09-10T21:00:40.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-09-10T21:00:40.883 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.6 seq 150323855390 2024-09-10T21:00:40.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:00:41.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855390 -lt 150323855390 2024-09-10T21:00:41.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:41.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691866 2024-09-10T21:00:41.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:41.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:00:41.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691866 2024-09-10T21:00:41.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:41.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691866 2024-09-10T21:00:41.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-09-10T21:00:41.162 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.7 seq 171798691866 2024-09-10T21:00:41.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:00:41.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691867 -lt 171798691866 2024-09-10T21:00:41.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:41.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528335 2024-09-10T21:00:41.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:41.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:00:41.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528335 2024-09-10T21:00:41.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:41.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528335 2024-09-10T21:00:41.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-09-10T21:00:41.458 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.8 seq 193273528335 2024-09-10T21:00:41.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:00:41.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528336 -lt 193273528335 2024-09-10T21:00:41.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:41.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364808 2024-09-10T21:00:41.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:41.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:00:41.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364808 2024-09-10T21:00:41.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:41.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364808 2024-09-10T21:00:41.740 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.9 seq 214748364808 2024-09-10T21:00:41.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-09-10T21:00:41.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:00:42.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364808 -lt 214748364808 2024-09-10T21:00:42.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:00:42.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:42.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:42.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:00:42.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:00:42.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:00:42.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:00:42.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:42.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:00:43.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:00:43.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:00:43.603 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:00:43.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:00:44.065 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:00:44.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:00:44.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:00:44.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:00:44.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T21:00:44.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T21:00:44.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:00:44.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:00:44.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:00:44.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:00:44.940 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:00:45.226 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:9 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:7' 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-10T21:00:45.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:00:45.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:00:45.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:00:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:00:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:00:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:00:45.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:00:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:00:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:00:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:00:45.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:00:47.575 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:00:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:00:48.094 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:00:48.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:00:48.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:00:48.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:00:48.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:00:48.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:00:48.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:00:48.124 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:00:48.120+0000 7fe393cb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:00:48.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:00:48.123+0000 7fe393cb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:00:48.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:00:48.124+0000 7fe393cb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:00:48.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:00:48.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:00:48.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:49.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:00:49.730 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:00:49.727+0000 7fe393cb2640 -1 Falling back to public interface 2024-09-10T21:00:49.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:50.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:00:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:00:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:00:51.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:00:51.384 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:00:51.378+0000 7fe393cb2640 -1 osd.3 68 log_to_monitors true 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:00:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/1450076371,v1:127.0.0.1:6827/1450076371] [v2:127.0.0.1:6828/1450076371,v1:127.0.0.1:6829/1450076371] exists,up a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:00:52.622 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:00:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:00:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:00:52.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:00:52.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:00:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:00:52.990 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:00:52.990 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:00:52.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:00:52.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:52.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:00:53.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836562 2024-09-10T21:00:53.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836562 2024-09-10T21:00:53.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562' 2024-09-10T21:00:53.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:00:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673029 2024-09-10T21:00:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673029 2024-09-10T21:00:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029' 2024-09-10T21:00:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:00:53.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-10T21:00:53.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-10T21:00:53.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503' 2024-09-10T21:00:53.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.311 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:00:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-10T21:00:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-10T21:00:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314' 2024-09-10T21:00:53.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:00:53.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182446 2024-09-10T21:00:53.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182446 2024-09-10T21:00:53.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446' 2024-09-10T21:00:53.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:00:53.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018919 2024-09-10T21:00:53.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018919 2024-09-10T21:00:53.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446 5-128849018919' 2024-09-10T21:00:53.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:00:53.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855394 2024-09-10T21:00:53.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855394 2024-09-10T21:00:53.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446 5-128849018919 6-150323855394' 2024-09-10T21:00:53.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:00:53.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691870 2024-09-10T21:00:53.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691870 2024-09-10T21:00:53.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446 5-128849018919 6-150323855394 7-171798691870' 2024-09-10T21:00:53.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:00:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528339 2024-09-10T21:00:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528339 2024-09-10T21:00:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446 5-128849018919 6-150323855394 7-171798691870 8-193273528339' 2024-09-10T21:00:53.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:00:53.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:00:54.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364812 2024-09-10T21:00:54.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364812 2024-09-10T21:00:54.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673029 2-64424509503 3-309237645314 4-107374182446 5-128849018919 6-150323855394 7-171798691870 8-193273528339 9-214748364812' 2024-09-10T21:00:54.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:54.078 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836562 2024-09-10T21:00:54.078 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:54.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:00:54.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836562 2024-09-10T21:00:54.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:54.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836562 2024-09-10T21:00:54.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-10T21:00:54.080 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836562 2024-09-10T21:00:54.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:00:54.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836562 2024-09-10T21:00:54.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:54.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673029 2024-09-10T21:00:54.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:54.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:00:54.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673029 2024-09-10T21:00:54.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:54.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673029 2024-09-10T21:00:54.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-09-10T21:00:54.361 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673029 2024-09-10T21:00:54.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:00:54.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673028 -lt 42949673029 2024-09-10T21:00:54.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:00:55.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:00:55.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:00:55.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673030 -lt 42949673029 2024-09-10T21:00:55.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:55.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-10T21:00:55.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:55.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:00:55.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-10T21:00:55.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:55.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-10T21:00:55.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-10T21:00:55.929 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509503 2024-09-10T21:00:55.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:00:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509503 2024-09-10T21:00:56.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:56.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645314 2024-09-10T21:00:56.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:56.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:00:56.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645314 2024-09-10T21:00:56.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-10T21:00:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-09-10T21:00:56.251 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 309237645314 2024-09-10T21:00:56.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:00:56.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-10T21:00:56.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:56.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182446 2024-09-10T21:00:56.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:56.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:00:56.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182446 2024-09-10T21:00:56.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182446 2024-09-10T21:00:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-09-10T21:00:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182446 2024-09-10T21:00:56.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:00:56.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182446 -lt 107374182446 2024-09-10T21:00:56.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:56.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018919 2024-09-10T21:00:56.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:56.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:00:56.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018919 2024-09-10T21:00:56.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:56.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018919 2024-09-10T21:00:56.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-09-10T21:00:56.852 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 128849018919 2024-09-10T21:00:56.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:00:57.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018919 -lt 128849018919 2024-09-10T21:00:57.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:57.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855394 2024-09-10T21:00:57.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:57.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:00:57.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855394 2024-09-10T21:00:57.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:57.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855394 2024-09-10T21:00:57.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-09-10T21:00:57.132 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855394 2024-09-10T21:00:57.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:00:57.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855395 -lt 150323855394 2024-09-10T21:00:57.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:57.422 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691870 2024-09-10T21:00:57.422 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:57.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:00:57.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691870 2024-09-10T21:00:57.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691870 2024-09-10T21:00:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-09-10T21:00:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691870 2024-09-10T21:00:57.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:00:57.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691871 -lt 171798691870 2024-09-10T21:00:57.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:57.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528339 2024-09-10T21:00:57.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:57.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:00:57.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528339 2024-09-10T21:00:57.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:57.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528339 2024-09-10T21:00:57.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-09-10T21:00:57.704 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528339 2024-09-10T21:00:57.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:00:57.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528340 -lt 193273528339 2024-09-10T21:00:57.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:00:57.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364812 2024-09-10T21:00:57.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:00:57.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:00:57.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364812 2024-09-10T21:00:57.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:00:57.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364812 2024-09-10T21:00:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-09-10T21:00:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 214748364812 2024-09-10T21:00:57.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:00:58.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364812 -lt 214748364812 2024-09-10T21:00:58.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:00:58.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:58.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:58.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:00:58.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:00:58.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:00:58.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:00:58.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:00:58.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:00:58.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:00:58.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:00:58.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:00:58.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:00:58.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:00:58.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:00:59.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:00:59.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:00:59.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:00:59.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T21:00:59.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:00:59.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:00:59.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:00:59.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:00:59.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:00:59.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:00:36.215133+0000 2024-09-10T21:00:59.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:01:00.040 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:01:00.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:00.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:00.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:00:36.215133+0000 '>' 2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:00.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:01:01.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:01:01.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:01.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:01.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:01.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:01.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:01.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:01.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:00:36.215133+0000 '>' 2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:01.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:01:02.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:01:02.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:02.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:02.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:02.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:02.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:02.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:02.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:00:36.215133+0000 '>' 2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:02.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:01:03.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:01:03.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:03.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:03.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:03.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:03.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:03.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:04.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:00.578684+0000 '>' 2024-09-10T21:00:36.215133+0000 2024-09-10T21:01:04.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:01:04.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:06.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:01:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:01:06.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:01:06.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:06.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:01:06.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:01:06.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:01:06.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:01:06.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:06.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:01:07.050 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T21:01:07.051 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:07.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:01:07.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:01:07.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:01:07.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:01:07.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:01:07.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.9 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:01:07.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:01:07.354 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:07.350+0000 7f3aff232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:07.518 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:07.514+0000 7f3aff232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:07.519 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:07.516+0000 7f3aff232640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:07.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:08.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:08.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:08.861+0000 7f3aff232640 -1 Falling back to public interface 2024-09-10T21:01:09.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:09.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:09.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:01:09.027 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:09.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:09.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:09.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:10.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:10.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:10.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:01:10.317 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:10.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:10.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:10.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:11.063 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:11.059+0000 7f3aff232640 -1 osd.9 73 log_to_monitors true 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:11.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:11.893 INFO:tasks.workunit.client.0.smithi181.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/3374409466,v1:127.0.0.1:6875/3374409466] [v2:127.0.0.1:6876/3374409466,v1:127.0.0.1:6877/3374409466] exists,up 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:01:11.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:01:11.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:01:11.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:01:11.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:01:12.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:01:12.382 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:12.382 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:12.382 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:12.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:01:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-10T21:01:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-10T21:01:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-10T21:01:12.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:12.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:01:12.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673034 2024-09-10T21:01:12.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673034 2024-09-10T21:01:12.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034' 2024-09-10T21:01:12.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:12.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:01:13.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509508 2024-09-10T21:01:13.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509508 2024-09-10T21:01:13.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508' 2024-09-10T21:01:13.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:01:13.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645319 2024-09-10T21:01:13.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645319 2024-09-10T21:01:13.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319' 2024-09-10T21:01:13.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:01:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182451 2024-09-10T21:01:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182451 2024-09-10T21:01:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451' 2024-09-10T21:01:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:01:13.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018924 2024-09-10T21:01:13.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018924 2024-09-10T21:01:13.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451 5-128849018924' 2024-09-10T21:01:13.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:01:13.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855399 2024-09-10T21:01:13.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855399 2024-09-10T21:01:13.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451 5-128849018924 6-150323855399' 2024-09-10T21:01:13.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:01:13.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691875 2024-09-10T21:01:13.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691875 2024-09-10T21:01:13.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451 5-128849018924 6-150323855399 7-171798691875' 2024-09-10T21:01:13.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:01:13.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528344 2024-09-10T21:01:13.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528344 2024-09-10T21:01:13.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451 5-128849018924 6-150323855399 7-171798691875 8-193273528344' 2024-09-10T21:01:13.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:13.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:01:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481794 2024-09-10T21:01:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481794 2024-09-10T21:01:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673034 2-64424509508 3-309237645319 4-107374182451 5-128849018924 6-150323855399 7-171798691875 8-193273528344 9-330712481794' 2024-09-10T21:01:13.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:13.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-10T21:01:13.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:13.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:01:13.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-10T21:01:13.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-10T21:01:13.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-10T21:01:13.964 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836567 2024-09-10T21:01:13.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:01:14.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-10T21:01:14.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:14.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673034 2024-09-10T21:01:14.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:14.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:01:14.241 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673034 2024-09-10T21:01:14.241 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:14.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673034 2024-09-10T21:01:14.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-09-10T21:01:14.242 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673034 2024-09-10T21:01:14.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:01:14.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673034 -lt 42949673034 2024-09-10T21:01:14.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:14.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509508 2024-09-10T21:01:14.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:14.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:01:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509508 2024-09-10T21:01:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:14.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509508 2024-09-10T21:01:14.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-09-10T21:01:14.522 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509508 2024-09-10T21:01:14.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:01:14.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509508 -lt 64424509508 2024-09-10T21:01:14.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:14.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645319 2024-09-10T21:01:14.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:14.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:01:14.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645319 2024-09-10T21:01:14.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:14.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645319 2024-09-10T21:01:14.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-09-10T21:01:14.801 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 309237645319 2024-09-10T21:01:14.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:01:15.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645318 -lt 309237645319 2024-09-10T21:01:15.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:01:16.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:01:16.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:01:16.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645319 -lt 309237645319 2024-09-10T21:01:16.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:16.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182451 2024-09-10T21:01:16.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:16.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:01:16.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182451 2024-09-10T21:01:16.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:16.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182451 2024-09-10T21:01:16.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-10T21:01:16.372 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182451 2024-09-10T21:01:16.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:01:16.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182451 -lt 107374182451 2024-09-10T21:01:16.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:16.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018924 2024-09-10T21:01:16.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:16.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:01:16.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018924 2024-09-10T21:01:16.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:16.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018924 2024-09-10T21:01:16.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-09-10T21:01:16.651 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 128849018924 2024-09-10T21:01:16.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:01:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018924 -lt 128849018924 2024-09-10T21:01:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:16.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855399 2024-09-10T21:01:16.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:01:16.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855399 2024-09-10T21:01:16.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855399 2024-09-10T21:01:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-09-10T21:01:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855399 2024-09-10T21:01:16.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:01:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855399 -lt 150323855399 2024-09-10T21:01:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:17.211 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691875 2024-09-10T21:01:17.211 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:17.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:01:17.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691875 2024-09-10T21:01:17.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:17.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691875 2024-09-10T21:01:17.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-09-10T21:01:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691875 2024-09-10T21:01:17.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:01:17.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691876 -lt 171798691875 2024-09-10T21:01:17.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:17.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528344 2024-09-10T21:01:17.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:17.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:01:17.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528344 2024-09-10T21:01:17.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:17.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528344 2024-09-10T21:01:17.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-09-10T21:01:17.513 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528344 2024-09-10T21:01:17.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:01:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528345 -lt 193273528344 2024-09-10T21:01:17.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:17.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481794 2024-09-10T21:01:17.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:17.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:01:17.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481794 2024-09-10T21:01:17.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481794 2024-09-10T21:01:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-09-10T21:01:17.793 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 330712481794 2024-09-10T21:01:17.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:01:18.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481795 -lt 330712481794 2024-09-10T21:01:18.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:01:18.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:18.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:01:18.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:01:18.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:01:18.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:01:18.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:18.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:01:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:01:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:01:19.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:01:19.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:01:19.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:01:19.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:01:19.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:01:19.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:01:19.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:01:19.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:01:19.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:01:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:01:21.674 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:01:22.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:01:22.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:01:22.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:01:22.198 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.5 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:01:22.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:01:22.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:22.223+0000 7f57c7844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:22.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:22.226+0000 7f57c7844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:22.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:22.227+0000 7f57c7844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:22.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:01:22.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:22.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:22.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:23.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:23.350+0000 7f57c7844640 -1 Falling back to public interface 2024-09-10T21:01:23.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:01:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:23.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:24.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:24.981+0000 7f57c7844640 -1 osd.5 79 log_to_monitors true 2024-09-10T21:01:25.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:25.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:25.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:01:25.056 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:25.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:25.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:01:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:26.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:26.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/2444442949,v1:127.0.0.1:6843/2444442949] [v2:127.0.0.1:6844/2444442949,v1:127.0.0.1:6845/2444442949] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:01:26.746 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:01:26.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:01:26.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:01:26.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:01:26.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:01:26.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:01:26.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:01:26.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:01:26.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:01:26.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:01:27.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:01:27.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836571 2024-09-10T21:01:27.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836571 2024-09-10T21:01:27.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571' 2024-09-10T21:01:27.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:01:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673038 2024-09-10T21:01:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673038 2024-09-10T21:01:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038' 2024-09-10T21:01:27.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:01:27.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-10T21:01:27.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-10T21:01:27.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512' 2024-09-10T21:01:27.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:01:27.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645323 2024-09-10T21:01:27.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645323 2024-09-10T21:01:27.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323' 2024-09-10T21:01:27.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:01:27.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182455 2024-09-10T21:01:27.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182455 2024-09-10T21:01:27.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455' 2024-09-10T21:01:27.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:01:27.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-10T21:01:27.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-10T21:01:27.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455 5-360777252866' 2024-09-10T21:01:27.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:27.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:01:28.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855403 2024-09-10T21:01:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855403 2024-09-10T21:01:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455 5-360777252866 6-150323855403' 2024-09-10T21:01:28.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:28.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:01:28.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-10T21:01:28.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-10T21:01:28.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455 5-360777252866 6-150323855403 7-171798691879' 2024-09-10T21:01:28.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:28.142 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:01:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528348 2024-09-10T21:01:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528348 2024-09-10T21:01:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455 5-360777252866 6-150323855403 7-171798691879 8-193273528348' 2024-09-10T21:01:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:28.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:01:28.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481798 2024-09-10T21:01:28.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481798 2024-09-10T21:01:28.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673038 2-64424509512 3-309237645323 4-107374182455 5-360777252866 6-150323855403 7-171798691879 8-193273528348 9-330712481798' 2024-09-10T21:01:28.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:28.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836571 2024-09-10T21:01:28.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:28.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:01:28.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836571 2024-09-10T21:01:28.352 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:28.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836571 2024-09-10T21:01:28.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-09-10T21:01:28.353 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836571 2024-09-10T21:01:28.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:01:28.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836570 -lt 21474836571 2024-09-10T21:01:28.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:01:29.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:01:29.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:01:29.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836571 2024-09-10T21:01:29.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:29.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673038 2024-09-10T21:01:29.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:29.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:01:29.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673038 2024-09-10T21:01:29.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673038 2024-09-10T21:01:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-10T21:01:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673038 2024-09-10T21:01:29.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:01:30.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673038 2024-09-10T21:01:30.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:30.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-10T21:01:30.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:30.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:01:30.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-10T21:01:30.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:30.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-10T21:01:30.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-10T21:01:30.205 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509512 2024-09-10T21:01:30.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:01:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-10T21:01:30.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:30.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645323 2024-09-10T21:01:30.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:01:30.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645323 2024-09-10T21:01:30.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:30.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645323 2024-09-10T21:01:30.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645323' 2024-09-10T21:01:30.487 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 309237645323 2024-09-10T21:01:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:01:30.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645323 -lt 309237645323 2024-09-10T21:01:30.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:30.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182455 2024-09-10T21:01:30.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:30.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:01:30.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182455 2024-09-10T21:01:30.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:30.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182455 2024-09-10T21:01:30.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-09-10T21:01:30.765 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182455 2024-09-10T21:01:30.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:01:31.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182455 -lt 107374182455 2024-09-10T21:01:31.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:31.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252866 2024-09-10T21:01:31.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:31.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:01:31.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252866 2024-09-10T21:01:31.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:31.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-10T21:01:31.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-09-10T21:01:31.045 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 360777252866 2024-09-10T21:01:31.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:01:31.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252867 -lt 360777252866 2024-09-10T21:01:31.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:31.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855403 2024-09-10T21:01:31.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:31.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:01:31.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855403 2024-09-10T21:01:31.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:31.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855403 2024-09-10T21:01:31.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-09-10T21:01:31.330 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855403 2024-09-10T21:01:31.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:01:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855403 -lt 150323855403 2024-09-10T21:01:31.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:31.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-10T21:01:31.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:31.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:01:31.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-10T21:01:31.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:31.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-10T21:01:31.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-10T21:01:31.612 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691879 2024-09-10T21:01:31.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:01:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-10T21:01:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:31.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528348 2024-09-10T21:01:31.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:31.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:01:31.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528348 2024-09-10T21:01:31.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528348 2024-09-10T21:01:31.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-09-10T21:01:31.996 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528348 2024-09-10T21:01:31.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:01:32.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528349 -lt 193273528348 2024-09-10T21:01:32.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:32.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481798 2024-09-10T21:01:32.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:32.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:01:32.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481798 2024-09-10T21:01:32.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:32.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481798 2024-09-10T21:01:32.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-09-10T21:01:32.317 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 330712481798 2024-09-10T21:01:32.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:01:32.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481798 -lt 330712481798 2024-09-10T21:01:32.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:01:32.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:32.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:01:32.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:01:33.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:01:33.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:01:33.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:33.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:01:33.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:01:33.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:01:33.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:01:33.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T21:01:33.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:33.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:34.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:01:34.421 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:01:34.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:34.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:01:34.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:34.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:34.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:00.578684+0000 '>' 2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:34.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:01:35.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:01:35.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:35.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:35.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:35.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:35.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:35.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:35.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:00.578684+0000 '>' 2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:35.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:01:37.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:01:37.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:01:37.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:01:37.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:01:37.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:01:37.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:01:37.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:01:37.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:34.620465+0000 '>' 2024-09-10T21:01:00.578684+0000 2024-09-10T21:01:37.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:01:37.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:01:39.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:01:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:01:39.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:01:39.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:01:39.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:01:39.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:01:39.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:01:39.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:01:39.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:01:39.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:01:39.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:39.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:01:40.171 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:01:40.172 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T21:01:40.172 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:01:40.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:01:40.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:01:40.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:01:40.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:01:40.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:01:40.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:01:40.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:01:40.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:01:40.442 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.9 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:01:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:01:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:40.467+0000 7fbf47c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:40.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:40.598+0000 7fbf47c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:40.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:40.599+0000 7fbf47c2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:40.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:01:40.808 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:01:40.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:40.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:41.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:41.945 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:41.941+0000 7fbf47c2c640 -1 Falling back to public interface 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:42.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:42.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:43.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:43.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:01:44.048+0000 7fbf47c2c640 -1 osd.9 85 log_to_monitors true 2024-09-10T21:01:44.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:44.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:44.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:01:44.740 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:44.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:44.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:osd.9 up in weight 1 up_from 89 up_thru 0 down_at 86 last_clean_interval [77,85) [v2:127.0.0.1:6874/3953903506,v1:127.0.0.1:6875/3953903506] [v2:127.0.0.1:6876/3953903506,v1:127.0.0.1:6877/3953903506] exists,up 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T21:01:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:01:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:01:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:01:45.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:01:45.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:01:45.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:01:45.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:01:45.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:45.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:01:46.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836576 2024-09-10T21:01:46.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836576 2024-09-10T21:01:46.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576' 2024-09-10T21:01:46.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:01:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673043 2024-09-10T21:01:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673043 2024-09-10T21:01:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043' 2024-09-10T21:01:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:01:46.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509516 2024-09-10T21:01:46.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509516 2024-09-10T21:01:46.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516' 2024-09-10T21:01:46.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:01:46.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645327 2024-09-10T21:01:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645327 2024-09-10T21:01:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327' 2024-09-10T21:01:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:01:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182459 2024-09-10T21:01:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182459 2024-09-10T21:01:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459' 2024-09-10T21:01:46.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:01:46.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252871 2024-09-10T21:01:46.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252871 2024-09-10T21:01:46.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459 5-360777252871' 2024-09-10T21:01:46.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:01:46.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855408 2024-09-10T21:01:46.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855408 2024-09-10T21:01:46.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459 5-360777252871 6-150323855408' 2024-09-10T21:01:46.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:46.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:01:47.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691884 2024-09-10T21:01:47.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-10T21:01:47.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459 5-360777252871 6-150323855408 7-171798691884' 2024-09-10T21:01:47.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:47.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:01:47.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528353 2024-09-10T21:01:47.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528353 2024-09-10T21:01:47.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459 5-360777252871 6-150323855408 7-171798691884 8-193273528353' 2024-09-10T21:01:47.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:01:47.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:01:47.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089346 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089346 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673043 2-64424509516 3-309237645327 4-107374182459 5-360777252871 6-150323855408 7-171798691884 8-193273528353 9-382252089346' 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836576 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:47.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:01:47.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836576 2024-09-10T21:01:47.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:47.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836576 2024-09-10T21:01:47.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-09-10T21:01:47.307 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836576 2024-09-10T21:01:47.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:01:47.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836576 -lt 21474836576 2024-09-10T21:01:47.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:47.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673043 2024-09-10T21:01:47.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:47.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:01:47.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673043 2024-09-10T21:01:47.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:47.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673043 2024-09-10T21:01:47.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-10T21:01:47.598 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673043 2024-09-10T21:01:47.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:01:47.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-10T21:01:47.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:47.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509516 2024-09-10T21:01:47.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:47.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:01:47.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509516 2024-09-10T21:01:47.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:47.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509516 2024-09-10T21:01:47.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-09-10T21:01:47.877 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509516 2024-09-10T21:01:47.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:01:48.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509516 2024-09-10T21:01:48.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:48.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645327 2024-09-10T21:01:48.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:48.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:01:48.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645327 2024-09-10T21:01:48.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:48.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645327 2024-09-10T21:01:48.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645327' 2024-09-10T21:01:48.159 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 309237645327 2024-09-10T21:01:48.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:01:48.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645328 -lt 309237645327 2024-09-10T21:01:48.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:48.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182459 2024-09-10T21:01:48.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:48.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:01:48.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182459 2024-09-10T21:01:48.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:48.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182459 2024-09-10T21:01:48.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-09-10T21:01:48.436 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182459 2024-09-10T21:01:48.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:01:48.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182460 -lt 107374182459 2024-09-10T21:01:48.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:48.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252871 2024-09-10T21:01:48.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:48.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:01:48.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252871 2024-09-10T21:01:48.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:48.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252871 2024-09-10T21:01:48.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-09-10T21:01:48.718 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 360777252871 2024-09-10T21:01:48.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:01:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252871 -lt 360777252871 2024-09-10T21:01:48.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:48.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855408 2024-09-10T21:01:48.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:48.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:01:48.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855408 2024-09-10T21:01:48.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:48.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855408 2024-09-10T21:01:48.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-09-10T21:01:48.998 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855408 2024-09-10T21:01:48.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:01:49.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855408 -lt 150323855408 2024-09-10T21:01:49.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:49.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691884 2024-09-10T21:01:49.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:49.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:01:49.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691884 2024-09-10T21:01:49.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:49.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-10T21:01:49.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-10T21:01:49.285 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691884 2024-09-10T21:01:49.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:01:49.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691884 -lt 171798691884 2024-09-10T21:01:49.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:49.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528353 2024-09-10T21:01:49.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:49.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:01:49.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528353 2024-09-10T21:01:49.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:49.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528353 2024-09-10T21:01:49.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2024-09-10T21:01:49.604 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528353 2024-09-10T21:01:49.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:01:49.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528353 -lt 193273528353 2024-09-10T21:01:49.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:01:49.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-382252089346 2024-09-10T21:01:49.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:01:49.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:01:49.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-382252089346 2024-09-10T21:01:49.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:01:49.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089346 2024-09-10T21:01:49.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 382252089346' 2024-09-10T21:01:49.884 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 382252089346 2024-09-10T21:01:49.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:01:50.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089346 -lt 382252089346 2024-09-10T21:01:50.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:01:50.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:50.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:50.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:01:50.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:01:50.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:01:50.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:01:50.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:50.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:01:51.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:01:51.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:01:51.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:01:51.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:01:51.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:01:51.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:01:51.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:01:51.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:01:51.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:01:51.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:01:51.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:01:51.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:01:51.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:01:51.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:01:51.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:01:51.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:01:51.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:01:51.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:01:51.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-10T21:01:51.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 209018"' 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 209018' 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:01:51.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 209020"' 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 209020' 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 209018 209020' 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 209018 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/209021: /' 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:01:51.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/209023: /' 2024-09-10T21:01:55.161 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:01:55.162 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:01:55.164 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:01:55.165 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:01:55.166 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:01:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:01:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:01:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:01:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:209023: start osd.9 2024-09-10T21:01:55.167 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:2209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:01:55.218 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:01:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:01:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:01:55.221 INFO:tasks.workunit.client.0.smithi181.stderr:209021: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:01:55.222 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:01:55.223 INFO:tasks.workunit.client.0.smithi181.stderr:209021: start osd.5 2024-09-10T21:01:55.223 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:01:59.752 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:01:59.752 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:01:59.752 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:01:59.752 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:01:59.752 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2024-09-10T21:01:55.193+0000 7f506a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2024-09-10T21:01:55.195+0000 7f506a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2024-09-10T21:01:55.196+0000 7f506a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 0 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.753 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 1 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2024-09-10T21:01:56.787+0000 7f506a62c640 -1 Falling back to public interface 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2 2024-09-10T21:01:59.754 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2024-09-10T21:01:58.234+0000 7f506a62c640 -1 osd.9 91 log_to_monitors true 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 3 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.755 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209023: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 92 last_cl09021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2024-09-10T21:01:55.246+0000 7f3aee619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.953 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2024-09-10T21:01:55.257+0000 7f3aee619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2024-09-10T21:01:55.258+0000 7f3aee619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 0 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2024-09-10T21:01:56.618+0000 7f3aee619640 -1 Falling back to public interface 2024-09-10T21:01:59.954 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 1 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:01:59.955 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2024-09-10T21:01:58.917+0000 7f3aee619640 -1 osd.5 91 log_to_monitors true 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 3 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:01:59.956 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersean_interval [89,91) [v2:127.0.0.1:6874/1876915542,v1:127.0.0.1:6875/1876915542] [v2:127.0.0.1:6876/1876915542,v1:127.0.0.1:6877/1876915542] exists,up 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:00.628 INFO:tasks.workunit.client.0.smithi181.stderr:209023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 1 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 2 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 3 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 4 2024-09-10T21:02:00.629 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 5 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 6 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 7 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 8 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: 9' 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836580 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836580 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580' 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673047 2024-09-10T21:02:00.630 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673047 2024-09-10T21:02:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047' 2024-09-10T21:02:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 4 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: osd.5 up in weight 1 up_from 98 up_thru 98 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6842/1218612302,v1:127.0.0.1:6843/1218612302] [v2:127.0.0.1:6844/1218612302,v1:127.0.0.1:6845/1218612302] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:209021: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 1 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 2 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 3 2024-09-10T21:02:02.055 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 4 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 5 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 6 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 7 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 8 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: 9' 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836581 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836581 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581' 2024-09-10T21:02:02.056 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.331 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:02.331 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520' 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645331 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645331 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331' 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182463 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182463 2024-09-10T21:02:02.332 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463' 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463 5-420906795010' 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855412 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855412 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463 5-420906795010 6-150323855412' 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691888 2024-09-10T21:02:02.333 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691888 2024-09-10T21:02:02.334 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463 5-420906795010 6-150323855412 7-171798691888' 2024-09-10T21:02:02.334 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:02.334 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone./home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673048 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673048 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048' 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509522 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509522 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522' 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645333 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645333 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333' 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182465 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182465 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465' 2024-09-10T21:02:03.360 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465 5-420906795011' 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855413 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855413 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465 5-420906795011 6-150323855413' 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.361 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528357 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528357 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463 5-420906795010 6-150323855412 7-171798691888 8-193273528357' 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-10T21:02:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673047 2-64424509520 3-309237645331 4-107374182463 5-420906795010 6-150323855412 7-171798691888 8-193273528357 9-408021893122' 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836580 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836580 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836580 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.0 seq 21474836580 2024-09-10T21:02:03.399 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673047 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673047 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673047 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.1 seq 42949673047 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673047 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:03.400 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-10T21:02:03.401 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:03.401 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:03.899 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.clpg_stats: seq=171798691889 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691889 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465 5-420906795011 6-150323855413 7-171798691889' 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528358 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528358 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465 5-420906795011 6-150323855413 7-171798691889 8-193273528358' 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893124 2024-09-10T21:02:03.900 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893124 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673048 2-64424509522 3-309237645333 4-107374182465 5-420906795011 6-150323855413 7-171798691889 8-193273528358 9-408021893124' 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836581 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836581 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836581 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.0 seq 21474836581 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-10T21:02:03.901 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673048 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673048 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673048 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-09-10T21:02:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.1 seq 42949673048 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_sient.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.2 seq 64424509520 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:04.321 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509520 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645331 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645331 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645331 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.3 seq 309237645331 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645333 -lt 309237645331 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182463 2024-09-10T21:02:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182463 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182463 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.4 seq 107374182463 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182465 -lt 107374182463 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-420906795010 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:04.323 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-420906795010 2024-09-10T21:02:04.324 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:04.324 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-10T21:02:05.144 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waitintats: ceph osd last-stat-seq 1 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673048 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509522 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509522 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509522 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.2 seq 64424509522 2024-09-10T21:02:05.145 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509522 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645333 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645333 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645333 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645333' 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.3 seq 309237645333 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645333 -lt 309237645333 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.146 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182465 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182465 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182465 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.4 seq 107374182465 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182465 -lt 107374182465 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.147 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-420906795011 2024-09-10T21:02:05.542 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/ceg osd.5 seq 420906795010' 2024-09-10T21:02:05.542 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.5 seq 420906795010 2024-09-10T21:02:05.542 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795010 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855412 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855412 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855412 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.6 seq 150323855412 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855412 2024-09-10T21:02:05.543 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691888 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691888 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691888 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.7 seq 171798691888 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691888 -lt 171798691888 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528357 2024-09-10T21:02:05.544 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528357 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528357 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.8 seq 193273528357 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:05.545 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528359 -lt 193273528357 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-420906795011 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 420906795011' 2024-09-10T21:02:06.059 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.5 seq 420906795011 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795011 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855413 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855413 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855413 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.6 seq 150323855413 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:06.060 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855413 -lt 150323855413 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691889 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691889 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691889 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.7 seq 171798691889 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691890 -lt 171798691889 2024-09-10T21:02:06.061 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528358 2024-09-10T21:02:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:06.062 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528358 2024-09-10T21:02:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cstats: for s in $seqs 2024-09-10T21:02:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-408021893122 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-408021893122 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: waiting osd.9 seq 408021893122 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893124 -lt 408021893122 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:209023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:06.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:07.684 INFO:tasks.workunit.client.0.smithi181.stderr:ut -d - -f 2 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528358 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.8 seq 193273528358 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528359 -lt 193273528358 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-408021893124 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-408021893124 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893124 2024-09-10T21:02:07.685 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 408021893124' 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: waiting osd.9 seq 408021893124 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893124 -lt 408021893124 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:07.686 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:209021: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 209020 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:02:07.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:02:07.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:07.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:07.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:08.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:08.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:02:08.503 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:02:08.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:08.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:02:08.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:08.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:34.620465+0000 '>' 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:08.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:09.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:09.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:09.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:09.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:09.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:09.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:09.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:10.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:34.620465+0000 '>' 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:10.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:11.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:11.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:11.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:11.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:11.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:11.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:11.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:11.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:34.620465+0000 '>' 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:11.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:12.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:12.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:12.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:12.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:12.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:12.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:12.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:12.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:01:34.620465+0000 '>' 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:12.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:13.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:13.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:13.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:13.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:13.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:13.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:09.237904+0000 '>' 2024-09-10T21:01:34.620465+0000 2024-09-10T21:02:13.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:02:13.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 212454"' 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 212454' 2024-09-10T21:02:15.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 212456"' 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 212456' 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 212454 212456' 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 212454 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/212457: /' 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:02:15.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/212459: /' 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:17.059 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: _ 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: hinfo_key 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: snapset 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:02:17.060 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:17.061 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:17.062 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:17.063 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:17.063 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:02:17.063 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:02:17.063 INFO:tasks.workunit.client.0.smithi181.stderr:212459: start osd.9 2024-09-10T21:02:17.113 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:17.114 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: _ 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: hinfo_key 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: snapset 2024-09-10T21:02:17.115 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:17.116 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:17.117 INFO:tasks.workunit.client.0.smithi181.stderr:212457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:17.118 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:17.119 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:17.119 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:17.119 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:02:17.119 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:02:17.119 INFO:tasks.workunit.client.0.smithi181.stderr:212457: start osd.5 2024-09-10T21:02:21.932 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:02:21.932 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:02:21.932 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2024-09-10T21:02:17.142+0000 7f4a41c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2024-09-10T21:02:17.179+0000 7f4a41c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2024-09-10T21:02:17.180+0000 7f4a41c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:21.933 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 0 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2024-09-10T21:02:18.540+0000 7f4a41c0b640 -1 Falling back to public interface 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 1 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2024-09-10T21:02:20.761+0000 7f4a41c0b640 -1 osd.5 100 log_to_monitors true 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 3 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212457: osd.5 up in weight 1 up_from 104 up_thru 98 down_very_ops 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:02:22.042 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2024-09-10T21:02:17.087+0000 7f70d8c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2024-09-10T21:02:17.179+0000 7f70d8c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2024-09-10T21:02:17.180+0000 7f70d8c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 0 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:22.043 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 1 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2024-09-10T21:02:19.023+0000 7f70d8c0d640 -1 Falling back to public interface 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 3 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:22.045 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:22.759 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2024-09-10T21:02:22.038+0000 7f70at 101 last_clean_interval [98,100) [v2:127.0.0.1:6842/778304988,v1:127.0.0.1:6843/778304988] [v2:127.0.0.1:6844/778304988,v1:127.0.0.1:6845/778304988] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T21:02:22.759 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:22.759 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:22.760 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 1 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 2 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 3 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 4 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 5 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 6 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 7 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 8 2024-09-10T21:02:22.761 INFO:tasks.workunit.client.0.smithi181.stderr:212457: 9' 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836586 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836586 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586' 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673053 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673053 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053' 2024-09-10T21:02:22.762 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527' 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645338 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645338 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338' 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.071 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182470 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182470 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470' 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598786 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598786 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470 5-446676598786' 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855418 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855418 2024-09-10T21:02:24.072 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470 5-446676598786 6-150323855418' 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691894 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691894 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691894' 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:24.073 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:25.048 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528363 2024-09-10T21:02:25.048 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528363 2024-09-10T21:02:25.048 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691894 8-193273528363' 2024-09-10T21:02:25.048 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:25.048 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467970 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467970 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673053 2-64424509527 3-309237645338 4-107374182470 5-446676598786 6-150323855418 7-171798691894 8-193273528363 9-463856467970' 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836586 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836586 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836586 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.0 seq 21474836586 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:25.049 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836586 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673053 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673053 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673053 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.1 seq 42949673053 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673053 -lt 42949673053 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-10T21:02:25.050 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:25.051 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:25.348 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephted8c0d640 -1 osd.9 100 log_to_monitors true 2024-09-10T21:02:25.348 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 4 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 5 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:02:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:212459: osd.9 up in weight 1 up_from 108 up_thru 0 down_at 101 last_clean_interval [95,100) [v2:127.0.0.1:6874/2674416399,v1:127.0.0.1:6875/2674416399] [v2:127.0.0.1:6876/2674416399,v1:127.0.0.1:6877/2674416399] exists,up 6ac4d393-7386-4fbb-a034-1696d4eeda08 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:25.350 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 1 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 2 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 3 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 4 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 5 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 6 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 7 2024-09-10T21:02:25.351 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 8 2024-09-10T21:02:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:212459: 9' 2024-09-10T21:02:25.352 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalonst/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.2 seq 64424509527 2024-09-10T21:02:26.322 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509527 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645338 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645338 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645338 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645338' 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.3 seq 309237645338 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645338 -lt 309237645338 2024-09-10T21:02:26.323 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182470 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182470 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182470 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.4 seq 107374182470 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182470 -lt 107374182470 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:26.324 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-446676598786 2024-09-10T21:02:26.325 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:26.325 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:26.325 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-446676598786 2024-09-10T21:02:26.325 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:26.325 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598786 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: ee/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836588 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836588 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588' 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673055 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673055 2024-09-10T21:02:26.480 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055' 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509528 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509528 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528' 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645339 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645339 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339' 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182471 2024-09-10T21:02:26.481 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182471 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471' 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598788 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598788 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471 5-446676598788' 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:26.482 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855cho 'waiting osd.5 seq 446676598786' 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.5 seq 446676598786 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598786 -lt 446676598786 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855418 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855418 2024-09-10T21:02:27.507 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855418 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.6 seq 150323855418 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855418 -lt 150323855418 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691894 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691894 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691894 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-09-10T21:02:27.508 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.7 seq 171798691894 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691895 -lt 171798691894 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528363 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528363 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528363 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.8 seq 193273528363 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:27.509 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528365 -lt 193273528363 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218420 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855420 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471 5-446676598788 6-150323855420' 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691896 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691896 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471 5-446676598788 6-150323855420 7-171798691896' 2024-09-10T21:02:28.442 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528365 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528365 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471 5-446676598788 6-150323855420 7-171798691896 8-193273528365' 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467971 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467971 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673055 2-64424509528 3-309237645339 4-107374182471 5-446676598788 6-150323855420 7-171798691896 8-193273528365 9-463856467971' 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836588 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:28.443 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836588 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836588 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.0 seq 21474836588 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836588 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:02:28.444 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/st: flush_pg_stats: for s in $seqs 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467970 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467970 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467970 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467970' 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: waiting osd.9 seq 463856467970 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:28.915 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467971 -lt 463856467970 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:28.916 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:29.343 INFO:tasks.workunit.client.0.smithi181.stderr:212457: andalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836588 -lt 21474836588 2024-09-10T21:02:29.343 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:29.343 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673055 2024-09-10T21:02:29.343 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673055 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673055 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.1 seq 42949673055 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673055 -lt 42949673055 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509528 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:29.344 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509528 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509528 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.2 seq 64424509528 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509528 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645339 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645339 2024-09-10T21:02:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645339 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645339' 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.3 seq 309237645339 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645340 -lt 309237645339 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182471 2024-09-10T21:02:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T21:02:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:30.214 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format jso212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182471 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182471 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.4 seq 107374182471 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182472 -lt 107374182471 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-446676598788 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:30.215 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-446676598788 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598788 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 446676598788' 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.5 seq 446676598788 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598788 -lt 446676598788 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855420 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855420 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855420 2024-09-10T21:02:30.216 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.6 seq 150323855420 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855420 -lt 150323855420 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691896 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691896 2024-09-10T21:02:30.217 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:31.494 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691896 2024-09-10T21:02:31.494 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-10T21:02:31.494 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.7 seq 171798691896 2024-09-10T21:02:31.494 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691896 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528365 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528365 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528365 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.8 seq 193273528365 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528365 -lt 193273528365 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:31.495 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467971 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467971 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467971 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467971' 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: waiting osd.9 seq 463856467971 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467972 -lt 463856467971 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:31.496 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:31.497 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:31.497 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:31.497 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:31.497 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:31.497 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0n pg dump pgs 2024-09-10T21:02:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:02:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:02:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T21:02:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:02:31.944 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:02:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:02:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:02:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:02:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:02:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T21:02:31.946 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:02:31.947 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:31.947 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:31.947 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:31.947 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:31.947 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:31.948 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:31.948 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:31.948 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:31.948 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:32.151 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:32.211 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:212459: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:_num_pgs: ceph --format json status 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:212457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:212457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 212456 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:02:32.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:02:32.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:02:32.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T21:02:32.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:02:32.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:02:32.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:02:32.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 215591"' 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 215591' 2024-09-10T21:02:32.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 215593"' 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 215593' 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 215591 215593' 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 215591 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/215594: /' 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:02:32.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/215596: /' 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:02:35.679 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:35.680 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:35.681 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:02:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:02:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:215594: start osd.3 2024-09-10T21:02:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:02:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:2215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:02:36.058 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:02:36.059 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:36.060 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:36.061 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:36.062 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: start osd.5 2024-09-10T21:02:36.063 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:02:40.443 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2024-09-10T21:02:35.706+0000 7f64e69c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2024-09-10T21:02:35.709+0000 7f64e69c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2024-09-10T21:02:35.710+0000 7f64e69c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:02:40.444 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 0 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2024-09-10T21:02:36.842+0000 7f64e69c5640 -1 Falling back to public interface 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 1 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:02:40.445 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2024-09-10T21:02:38.491+0000 7f64e69c5640 -1 osd.3 110 log_to_monitors true 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 3 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:02:40.646 INFO:tasks.workunit.client.0.smithi181.stderr:215594: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 111 la15596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:02:40.646 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:02:40.646 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:02:40.646 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:02:40.646 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2024-09-10T21:02:36.084+0000 7f03ac216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2024-09-10T21:02:36.116+0000 7f03ac216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2024-09-10T21:02:36.117+0000 7f03ac216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 0 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:40.647 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 1 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2024-09-10T21:02:37.737+0000 7f03ac216640 -1 Falling back to public interface 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:02:40.648 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2024-09-10T21:02:39.669+0000 7f03ac216640 -1 osd.5 110 log_to_monitors true 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 3 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:40.649 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:41.248 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperst_clean_interval [72,110) [v2:127.0.0.1:6826/792746345,v1:127.0.0.1:6827/792746345] [v2:127.0.0.1:6828/792746345,v1:127.0.0.1:6829/792746345] exists,up a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T21:02:41.248 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:41.248 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:41.249 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 1 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 2 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 3 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 4 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 5 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 6 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 7 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 8 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: 9' 2024-09-10T21:02:41.250 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673059 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673059 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059' 2024-09-10T21:02:41.251 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532' 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271746 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271746 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746' 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182475 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182475 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475' 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:42.758 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173634 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173634 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475 5-502511173634' 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855424 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855424 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475 5-502511173634 6-150323855424' 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691900 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691900 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475 5-502511173634 6-150323855424 7-171798691900' 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.759 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clons.sh:976: wait_for_osd: sleep 1 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 4 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: osd.5 up in weight 1 up_from 117 up_thru 117 down_at 111 last_clean_interval [104,110) [v2:127.0.0.1:6842/2819353829,v1:127.0.0.1:6843/2819353829] [v2:127.0.0.1:6844/2819353829,v1:127.0.0.1:6845/2819353829] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:02:42.760 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:02:42.761 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 1 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 2 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 3 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 4 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 5 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 6 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 7 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 8 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: 9' 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:02:42.762 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:42.763 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:02:42.763 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836593 2024-09-10T21:02:42.763 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836593 2024-09-10T21:02:42.763 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593' 2024-09-10T21:02:42.763 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.604 INFO:tasks.workunit.client.0.smithi181.stderr:215e.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528369 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528369 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475 5-502511173634 6-150323855424 7-171798691900 8-193273528369' 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467975 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467975 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673059 2-64424509532 3-489626271746 4-107374182475 5-502511173634 6-150323855424 7-171798691900 8-193273528369 9-463856467975' 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:43.605 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.0 seq 21474836592 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673059 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673059 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673059 2024-09-10T21:02:43.606 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.1 seq 42949673059 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673059 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:02:43.717 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673060 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673060 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060' 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509534 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509534 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534' 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271747 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271747 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747' 2024-09-10T21:02:43.718 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182477 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182477 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477' 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173635 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173635 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477 5-502511173635' 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855425 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855425 2024-09-10T21:02:43.719 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477 5-502511173635 6-150323855425' 2024-09-10T21:02:43.720 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:43.720 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691901 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691901 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477 5-502511173635 6-150323855425 7-171798691901' 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:02:44.282 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528370 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528370 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477 5-502511173635 6-150323855425 7-171798691901 8-193273528370' 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467977 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467977 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673060 2-64424509534 3-489626271747 4-107374182477 5-502511173635 6-150323855425 7-171798691901 8-193273528370 9-463856467977' 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836593 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836593 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.283 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836593 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.0 seq 21474836593 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836593 -lt 21474836593 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673060 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673060 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673060 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-09-10T21:02:44.284 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.1 seq 42949673060 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flusclient.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.2 seq 64424509532 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509534 -lt 64424509532 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:44.500 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-489626271746 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-489626271746 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271746 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.3 seq 489626271746 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271746 -lt 489626271746 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182475 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:44.501 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182475 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182475 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.4 seq 107374182475 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182476 -lt 107374182475 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-502511173634 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-502511173634 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173634 2024-09-10T21:02:45.752 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 502511173634' 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.5 seq 502511173634 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173634 -lt 502511173634 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855424 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855424 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855424 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.6 seq 150323855424 2024-09-10T21:02:45.753 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855424 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691900 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691900 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691900 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.7 seq 171798691900 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691902 -lt 171798691900 2024-09-10T21:02:45.754 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528369 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528369 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528369 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.8 seq 193273528369 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:45.755 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528369 2024-09-10T21:02:46.481 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_ph_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:02:46.481 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673060 2024-09-10T21:02:46.481 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509534 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509534 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509534 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.2 seq 64424509534 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509534 -lt 64424509534 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:46.482 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-489626271747 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-489626271747 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271747 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.3 seq 489626271747 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271746 -lt 489626271747 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:02:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271748 -lt 489626271747 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182477 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182477 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182477 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-09-10T21:02:46.484 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.4 seq 107374182477 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephteg_stats: for s in $seqs 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467975 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467975 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:47.038 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467975 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467975' 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: waiting osd.9 seq 463856467975 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467977 -lt 463856467975 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:47.039 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:215594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:47.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 215593 2024-09-10T21:02:47.726 INFO:tasks.workunit.client.0.smithi181.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:02:47.726 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182477 -lt 107374182477 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-502511173635 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-502511173635 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173635 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 502511173635' 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.5 seq 502511173635 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173635 -lt 502511173635 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:47.727 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855425 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855425 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855425 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.6 seq 150323855425 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855426 -lt 150323855425 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691901 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:47.728 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691901 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691901 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.7 seq 171798691901 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691902 -lt 171798691901 2024-09-10T21:02:47.729 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528370 2024-09-10T21:02:48.909 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528370 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528370 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.8 seq 193273528370 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528370 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467977 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:02:48.910 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467977 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467977 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467977' 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: waiting osd.9 seq 463856467977 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467977 -lt 463856467977 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:02:48.911 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:02:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:02:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:02:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:02:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:02:49.254 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:215596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:215596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:02:49.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:02:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:02:49.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:49.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:49.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:49.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:49.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:02:50.071 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:50.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:50.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:50.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:50.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:50.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:09.237904+0000 '>' 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:50.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:51.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:51.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:51.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:51.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:51.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:51.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:51.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:51.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:09.237904+0000 '>' 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:51.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:52.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:52.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:52.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:52.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:52.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:52.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:52.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:52.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:09.237904+0000 '>' 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:52.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:53.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:53.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:53.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:53.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:53.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:53.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:53.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:09.237904+0000 '>' 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:02:55.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:02:55.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:02:55.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:02:55.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:02:55.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:02:55.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:02:55.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:02:55.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:02:50.591925+0000 '>' 2024-09-10T21:02:09.237904+0000 2024-09-10T21:02:55.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:02:55.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 219049"' 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 219049' 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:57.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 219050"' 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 219050' 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 219049 219050' 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 219049 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/219052: /' 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:57.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/219054: /' 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:02:58.633 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-10T21:02:58.634 INFO:tasks.workunit.client.0.smithi181.stderr:219054: _ 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: hinfo_key 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: snapset 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:58.635 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:58.636 INFO:tasks.workunit.client.0.smithi181.stderr:219054: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:58.637 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:58.638 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:02:58.638 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:02:58.638 INFO:tasks.workunit.client.0.smithi181.stderr:219054: start osd.5 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:02:58.676 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.677 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: _ 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: hinfo_key 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: snapset 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:02:58.678 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:02:58.679 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:02:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:02:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:02:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:02:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:02:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:02:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:219052: start osd.3 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2024-09-10T21:02:58.704+0000 7f7c5734b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2024-09-10T21:02:58.810+0000 7f7c5734b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2024-09-10T21:02:58.811+0000 7f7c5734b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:02.931 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 0 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:02.932 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2024-09-10T21:03:00.168+0000 7f7c5734b640 -1 Falling back to public interface 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 1 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:03:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2024-09-10T21:03:02.303+0000 7f7c5734b640 -1 osd.3 119 log_to_monitors true 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 3 2024-09-10T21:03:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:03.754 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpervery_ops 2024-09-10T21:03:03.754 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:03:03.754 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:03:03.754 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2024-09-10T21:02:58.663+0000 7fb42fe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2024-09-10T21:02:58.810+0000 7fb42fe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2024-09-10T21:02:58.811+0000 7fb42fe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:03:03.755 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 0 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 1 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:03.756 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2024-09-10T21:03:00.661+0000 7fb42fe0b640 -1 Falling back to public interface 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:03.757 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:03:03.758 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 3 2024-09-10T21:03:03.758 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:03.758 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:03.758 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2024-09-10T21:03:03.750+0000 7fb4s.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:03:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:219052: osd.3 up in weight 1 up_from 124 up_thru 117 down_at 120 last_clean_interval [114,119) [v2:127.0.0.1:6826/3302964143,v1:127.0.0.1:6827/3302964143] [v2:127.0.0.1:6828/3302964143,v1:127.0.0.1:6829/3302964143] exists,up a0e4d625-7d36-4c91-9851-31e1f8a00a28 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:03:04.351 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 1 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 2 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 3 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 4 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 5 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 6 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 7 2024-09-10T21:03:04.352 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 8 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: 9' 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836598 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836598 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598' 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673066 2024-09-10T21:03:04.353 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673066 2024-09-10T21:03:04.354 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066' 2024-09-10T21:03:05.618 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509539 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509539 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539' 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944706 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944706 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706' 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.619 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182482 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182482 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482' 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=2 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 2 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482 5-2' 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:03:05.620 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855430 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855430 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482 5-2 6-150323855430' 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691907 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691907 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482 5-2 6-150323855430 7-171798691907' 2024-09-10T21:03:05.621 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:06.716 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:03:06.716 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528376 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528376 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482 5-2 6-150323855430 7-171798691907 8-193273528376' 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467982 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467982 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673066 2-64424509539 3-532575944706 4-107374182482 5-2 6-150323855430 7-171798691907 8-193273528376 9-463856467982' 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836598 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836598 2024-09-10T21:03:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836598 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.0 seq 21474836598 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836598 -lt 21474836598 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673066 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673066 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673066 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-09-10T21:03:06.718 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.1 seq 42949673066 2024-09-10T21:03:06.719 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:03:06.719 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673066 -lt 42949673066 2024-09-10T21:03:06.719 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:06.719 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509539 2024-09-10T21:03:06.719 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2fe0b640 -1 osd.5 119 log_to_monitors true 2024-09-10T21:03:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 4 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 5 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: osd.5 up in weight 1 up_from 127 up_thru 127 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6842/4044130069,v1:127.0.0.1:6843/4044130069] [v2:127.0.0.1:6844/4044130069,v1:127.0.0.1:6845/4044130069] exists,up 83f9aad6-6a3b-488c-b138-3e6cb1c1bb92 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:03:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:03:07.019 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 1 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 2 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 3 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 4 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 5 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 6 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 7 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 8 2024-09-10T21:03:07.020 INFO:tasks.workunit.client.0.smithi181.stderr:219054: 9' 2024-09-10T21:03:07.021 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:03:07.662 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standa2 2024-09-10T21:03:07.662 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509539 2024-09-10T21:03:07.662 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:07.662 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509539 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.2 seq 64424509539 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509539 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-532575944706 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-532575944706 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944706 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 532575944706' 2024-09-10T21:03:07.663 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.3 seq 532575944706 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944706 -lt 532575944706 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182482 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182482 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182482 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.4 seq 107374182482 2024-09-10T21:03:07.664 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182483 -lt 107374182482 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-2 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-2 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:07.665 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=2 2024-09-10T21:03:07.839 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echolone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.839 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:03:07.839 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836600 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836600 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600' 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067' 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509541 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509541 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541' 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.840 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944707 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944707 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707' 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182484 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182484 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484' 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=545460846595 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 545460846595 2024-09-10T21:03:07.841 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484 5-545460846595' 2024-09-10T21:03:07.842 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:07.842 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:03:08.454 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855432 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855432 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484 5-545460846595 6-150323855432' 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691908 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691908 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484 5-545460846595 6-150323855432 7-171798691908' 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528377 2024-09-10T21:03:08.455 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528377 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484 5-545460846595 6-150323855432 7-171798691908 8-193273528377' 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467983 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467983 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673067 2-64424509541 3-532575944707 4-107374182484 5-545460846595 6-150323855432 7-171798691908 8-193273528377 9-463856467983' 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836600 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836600 2024-09-10T21:03:08.456 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836600 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.0 seq 21474836600 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836600 -lt 21474836600 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-10T21:03:08.457 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/s 'waiting osd.5 seq 2' 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.5 seq 2 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 2 -lt 2 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855430 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855430 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855430 2024-09-10T21:03:08.890 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.6 seq 150323855430 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855431 -lt 150323855430 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691907 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691907 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691907 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.7 seq 171798691907 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:03:08.891 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691907 -lt 171798691907 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528376 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528376 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528376 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.8 seq 193273528376 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528376 -lt 193273528376 2024-09-10T21:03:08.892 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:10.203 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467982 2024-09-10T21:03:10.203 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467982 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467982 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467982' 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: waiting osd.9 seq 463856467982 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467982 -lt 463856467982 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:03:10.204 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:03:10.205 INFO:tasks.workunit.client.0.smithi181.stderr:219052: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:03:10.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:03:10.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:03:10.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 219050 2024-09-10T21:03:10.666 INFO:tasks.workunit.client.0.smithi181.stderr:tandalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:03:10.666 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-10T21:03:10.666 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:10.666 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-10T21:03:10.666 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.1 seq 42949673067 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673066 -lt 42949673067 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673068 -lt 42949673067 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509541 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509541 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:10.667 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509541 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.2 seq 64424509541 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509541 -lt 64424509541 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-532575944707 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-532575944707 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944707 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 532575944707' 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.3 seq 532575944707 2024-09-10T21:03:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:03:10.669 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944708 -lt 532575944707 2024-09-10T21:03:10.669 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:10.669 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182484 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182484 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182484 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-09-10T21:03:11.530 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.4 seq 107374182484 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182484 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-545460846595 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-545460846595 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=545460846595 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 545460846595' 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.5 seq 545460846595 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:03:11.531 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 545460846595 -lt 545460846595 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855432 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855432 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855432 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.6 seq 150323855432 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855433 -lt 150323855432 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:11.532 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:11.533 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691908 2024-09-10T21:03:11.533 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:03:11.533 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691908 2024-09-10T21:03:11.533 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:12.715 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691908 2024-09-10T21:03:12.715 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-09-10T21:03:12.715 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.7 seq 171798691908 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691909 -lt 171798691908 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528377 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528377 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528377 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.8 seq 193273528377 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:03:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528378 -lt 193273528377 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467983 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467983 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467983 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467983' 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: waiting osd.9 seq 463856467983 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467984 -lt 463856467983 2024-09-10T21:03:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:03:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:219054: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:03:13.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:03:13.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:03:13.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:03:13.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T21:03:13.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:03:13.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:03:13.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:03:13.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:03:13.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:03:13.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:03:13.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:03:13.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:03:13.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:03:13.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:03:13.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:03:13.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:03:13.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:03:13.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:03:13.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:03:13.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:03:13.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:03:13.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:03:13.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:03:13.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:03:13.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:03:13.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:03:13.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:13.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:13.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:03:13.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:03:13.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:03:13.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:03:13.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:03:13.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:03:13.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:03:13.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:03:13.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:03:13.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:03:13.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:03:13.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:03:13.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:03:13.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:03:13.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:03:13.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:03:13.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:03:13.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:03:13.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:13.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:13.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:03:13.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:03:13.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:03:13.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:03:13.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:03:13.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:13.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:13.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:03:13.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:03:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:03:13.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:03:13.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:03:13.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:14.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:03:14.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:03:14.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:03:14.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:03:14.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:03:14.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:03:14.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:03:14.276 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:03:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:03:14.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:03:14.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:03:14.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:03:14.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:03:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-10T21:03:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:03:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:03:14.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:03:14.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:03:14.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:03:14.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:03:15.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:03:15.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:03:15.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:03:15.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:03:15.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:15.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:15.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:15.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:03:15.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:03:15.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-10T21:03:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:03:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-10T21:03:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:03:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:03:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:03:15.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:03:15.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:03:15.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:03:15.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77e65cda-4f18-404d-b87c-e182b9f0a902 2024-09-10T21:03:15.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 77e65cda-4f18-404d-b87c-e182b9f0a902' 2024-09-10T21:03:15.281 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 77e65cda-4f18-404d-b87c-e182b9f0a902 2024-09-10T21:03:15.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:03:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATtOBm6mhxERAA00qJVzS/8/uVyOTIZq2/tw== 2024-09-10T21:03:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATtOBm6mhxERAA00qJVzS/8/uVyOTIZq2/tw=="}' 2024-09-10T21:03:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77e65cda-4f18-404d-b87c-e182b9f0a902 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:03:15.499 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:03:15.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:03:15.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATtOBm6mhxERAA00qJVzS/8/uVyOTIZq2/tw== --osd-uuid 77e65cda-4f18-404d-b87c-e182b9f0a902 2024-09-10T21:03:15.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:15.535+0000 7ff9f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:15.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:15.546+0000 7ff9f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:15.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:15.547+0000 7ff9f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:15.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:15.547+0000 7ff9f4e0f640 -1 bdev(0x55cd9e2f5c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:03:15.551 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:15.547+0000 7ff9f4e0f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:03:21.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:03:21.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:03:21.281 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:03:21.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:03:21.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:03:21.639 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:03:21.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:03:21.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:03:21.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:03:21.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:03:21.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:03:21.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:21.664+0000 7f7f650e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:21.670 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:21.666+0000 7f7f650e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:21.671 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:21.668+0000 7f7f650e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:03:21.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:21.920 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:03:21.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:03:21.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:21.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:22.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:23.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:23.033+0000 7f7f650e4640 -1 Falling back to public interface 2024-09-10T21:03:23.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:23.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:23.195 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:03:23.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:03:23.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:23.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:23.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:24.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:24.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:24.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:03:24.487 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:03:24.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:24.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:24.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:24.938 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:24.934+0000 7f7f650e4640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:03:25.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:25.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:25.762 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:03:25.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:03:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:26.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:27.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:27.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:27.044 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:03:27.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:03:27.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:27.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:27.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:28.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:28.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:28.332 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:03:28.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:03:28.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:28.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:28.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:29.620 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:03:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:03:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:30.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:31.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:31.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:31.022 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:03:31.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:03:31.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:31.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:31.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:32.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:32.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:32.309 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:03:32.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:03:32.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:32.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:32.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:33.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:33.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:33.598 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:03:33.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:03:33.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:33.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:33.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:34.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:34.972 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:03:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:03:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:34.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:35.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:36.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:36.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:36.261 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:03:36.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:03:36.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:36.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:36.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:37.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:37.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:37.794 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:03:37.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:03:37.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:37.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:38.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:39.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:39.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:39.228 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:03:39.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:03:39.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:39.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:39.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:40.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:40.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:40.899 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:03:40.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:03:40.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:40.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:42.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:42.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:42.369 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:03:42.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:03:42.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:42.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:43.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:43.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:43.898 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:03:43.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:03:43.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:43.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:44.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:45.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:45.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:45.200 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:03:45.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:03:45.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:45.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:46.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:46.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:46.574 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:03:46.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:03:46.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:46.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:03:46.911 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3230279343,v1:127.0.0.1:6803/3230279343] [v2:127.0.0.1:6804/3230279343,v1:127.0.0.1:6805/3230279343] exists,up 77e65cda-4f18-404d-b87c-e182b9f0a902 2024-09-10T21:03:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:03:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:03:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:03:46.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd5ce41b-7c06-4d3f-a570-c151cea611c8 2024-09-10T21:03:46.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fd5ce41b-7c06-4d3f-a570-c151cea611c8' 2024-09-10T21:03:46.919 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 fd5ce41b-7c06-4d3f-a570-c151cea611c8 2024-09-10T21:03:46.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:03:46.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAytOBmtV1vNxAAgwrgNX/NXNKuZSATtHAjrA== 2024-09-10T21:03:46.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAytOBmtV1vNxAAgwrgNX/NXNKuZSATtHAjrA=="}' 2024-09-10T21:03:46.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd5ce41b-7c06-4d3f-a570-c151cea611c8 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:03:47.332 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:03:47.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:03:47.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAytOBmtV1vNxAAgwrgNX/NXNKuZSATtHAjrA== --osd-uuid fd5ce41b-7c06-4d3f-a570-c151cea611c8 2024-09-10T21:03:47.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:47.370+0000 7ffb80828640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:47.420 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:47.416+0000 7ffb80828640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:47.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:47.417+0000 7ffb80828640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:47.421 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:47.418+0000 7ffb80828640 -1 bdev(0x55bcec0b5c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:03:47.422 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:47.418+0000 7ffb80828640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:03:53.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:03:53.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:03:53.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:03:53.419 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:03:53.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:03:53.776 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:03:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:03:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:03:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:03:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:03:53.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:03:53.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:53.800+0000 7ff47e02f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:53.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:53.802+0000 7ff47e02f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:53.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:53.803+0000 7ff47e02f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:03:54.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:54.053 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:03:54.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:03:54.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:54.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:03:54.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:55.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:55.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:55.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:03:55.329 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:03:55.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:55.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:03:55.417 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:55.414+0000 7ff47e02f640 -1 Falling back to public interface 2024-09-10T21:03:55.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:56.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:56.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:56.617 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:03:56.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:03:56.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:56.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:03:56.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:03:56.752+0000 7ff47e02f640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:03:56.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:57.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:57.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:57.950 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:03:57.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:03:57.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:57.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:03:58.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:03:59.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:03:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:03:59.266 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:03:59.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:03:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:03:59.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:03:59.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:00.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:00.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:00.621 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:04:00.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:04:00.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:00.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:00.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:01.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:01.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:01.910 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:04:01.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:04:01.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:01.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:02.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:03.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:03.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:03.198 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:04:03.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:04:03.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:03.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:03.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:04.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:04.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:04.511 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:04:04.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:04:04.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:04.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:04.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:05.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:05.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:05.798 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:04:05.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:04:05.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:05.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:06.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:07.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:07.087 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:04:07.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:07.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:04:07.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:07.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:07.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:08.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:08.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:08.374 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:04:08.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:04:08.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:08.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:09.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:10.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:10.042 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:04:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:04:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:10.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:10.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:11.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:11.765 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:04:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:04:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:11.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:12.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:13.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:13.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:13.504 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:04:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:04:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:14.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:15.082 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:04:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:04:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:15.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:16.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:16.618 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:04:16.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:16.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:04:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:16.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:17.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:18.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:18.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:18.514 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:04:18.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:04:18.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:18.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:18.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:19.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:19.972 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:04:19.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:19.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:04:19.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:19.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:04:20.295 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3114925417,v1:127.0.0.1:6811/3114925417] [v2:127.0.0.1:6812/3114925417,v1:127.0.0.1:6813/3114925417] exists,up fd5ce41b-7c06-4d3f-a570-c151cea611c8 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:04:20.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:04:20.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:04:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:04:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=682b9e20-31ce-4c59-abbd-a652fbf08e3b 2024-09-10T21:04:20.304 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 682b9e20-31ce-4c59-abbd-a652fbf08e3b 2024-09-10T21:04:20.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 682b9e20-31ce-4c59-abbd-a652fbf08e3b' 2024-09-10T21:04:20.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:04:20.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUtOBm8Lu8EhAAFnivFu5i+hMwV5RYMA0YEA== 2024-09-10T21:04:20.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUtOBm8Lu8EhAAFnivFu5i+hMwV5RYMA0YEA=="}' 2024-09-10T21:04:20.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 682b9e20-31ce-4c59-abbd-a652fbf08e3b -i td/osd-scrub-repair/2/new.json 2024-09-10T21:04:20.663 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:04:20.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:04:20.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUtOBm8Lu8EhAAFnivFu5i+hMwV5RYMA0YEA== --osd-uuid 682b9e20-31ce-4c59-abbd-a652fbf08e3b 2024-09-10T21:04:20.704 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:20.700+0000 7f0d97158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:20.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:20.743+0000 7f0d97158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:20.744+0000 7f0d97158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:20.744+0000 7f0d97158640 -1 bdev(0x559e2eee8800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:04:20.749 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:20.744+0000 7f0d97158640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:04:26.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:04:26.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:04:26.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:04:26.907 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:04:26.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:04:27.271 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:04:27.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:04:27.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:04:27.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:04:27.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:04:27.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:04:27.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:27.295+0000 7ff74c9a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:27.296+0000 7ff74c9a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:27.297+0000 7ff74c9a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:27.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:04:27.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:04:27.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:04:27.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:27.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:27.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:28.682 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:28.678+0000 7ff74c9a2640 -1 Falling back to public interface 2024-09-10T21:04:28.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:28.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:28.832 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:04:28.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:04:28.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:28.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:29.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:30.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:30.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:30.121 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:04:30.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:04:30.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:30.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:30.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:30.266+0000 7ff74c9a2640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:04:30.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:31.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:31.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:31.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:04:31.472 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:04:31.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:31.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:31.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:32.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:32.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:32.760 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:04:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:04:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:33.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:34.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:34.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:34.144 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:04:34.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:04:34.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:34.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:34.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:35.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:35.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:35.431 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:04:35.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:04:35.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:35.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:35.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:36.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:36.827 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:04:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:04:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:36.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:37.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:38.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:38.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:38.113 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:04:38.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:04:38.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:38.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:38.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:39.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:39.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:39.457 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:04:39.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:04:39.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:39.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:39.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:40.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:40.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:40.827 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:04:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:04:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:40.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:41.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:42.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:42.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:42.118 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:04:42.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:04:42.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:42.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:42.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:43.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:43.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:43.405 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:04:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:04:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:43.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:44.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:45.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:45.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:45.307 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:04:45.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:04:45.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:45.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:45.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:46.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:46.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:46.709 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:04:46.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:04:46.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:46.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:47.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:48.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:48.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:48.420 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:04:48.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:04:48.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:48.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:49.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:50.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:50.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:50.023 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:04:50.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:04:50.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:50.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:50.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:51.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:51.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:51.527 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:04:51.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:04:51.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:51.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:52.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:04:53.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:04:53.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:04:53.550 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:04:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:04:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:04:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:04:53.843 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3711158399,v1:127.0.0.1:6819/3711158399] [v2:127.0.0.1:6820/3711158399,v1:127.0.0.1:6821/3711158399] exists,up 682b9e20-31ce-4c59-abbd-a652fbf08e3b 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:04:53.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:04:53.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:04:53.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:04:53.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1852b6c3-b374-40ec-b76e-f9ab64ef4e75' 2024-09-10T21:04:53.851 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:04:53.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:04:53.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1tOBmXuVZMxAAhAt5d2eBAi5HJlG/61WO5A== 2024-09-10T21:04:53.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1tOBmXuVZMxAAhAt5d2eBAi5HJlG/61WO5A=="}' 2024-09-10T21:04:53.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 -i td/osd-scrub-repair/3/new.json 2024-09-10T21:04:54.196 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:04:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T21:04:54.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1tOBmXuVZMxAAhAt5d2eBAi5HJlG/61WO5A== --osd-uuid 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:04:54.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:54.233+0000 7ff2a9a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:54.280 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:54.276+0000 7ff2a9a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:54.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:54.277+0000 7ff2a9a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:04:54.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:54.278+0000 7ff2a9a1c640 -1 bdev(0x55793ae51c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T21:04:54.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:04:54.278+0000 7ff2a9a1c640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T21:05:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T21:05:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:05:00.614 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T21:05:00.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T21:05:00.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:05:00.978 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T21:05:00.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T21:05:00.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:05:00.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:05:00.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:05:00.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:05:01.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:01.001+0000 7fc30e334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:01.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:01.003+0000 7fc30e334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:01.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:01.004+0000 7fc30e334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T21:05:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:05:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:05:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:05:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:05:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:05:01.284 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:05:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:01.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:02.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:02.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:02.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:05:02.561 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:05:02.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:02.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:02.619 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:02.616+0000 7fc30e334640 -1 Falling back to public interface 2024-09-10T21:05:02.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:03.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:03.842 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:05:03.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:03.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:05:03.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:03.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:04.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:04.255 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:04.251+0000 7fc30e334640 -1 osd.3 0 log_to_monitors true 2024-09-10T21:05:05.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:05.123 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:05:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:05:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:05.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:05.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:06.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:06.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:06.454 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:05:06.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:05:06.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:06.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:06.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:07.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:07.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:07.746 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:05:07.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:05:07.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:07.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:08.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:09.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:09.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:09.146 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:05:09.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:05:09.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:09.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:09.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:10.434 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:05:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:05:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:10.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:11.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:11.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:11.725 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:05:11.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:05:11.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:11.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:12.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:13.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:13.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:13.119 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:05:13.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:05:13.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:13.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:13.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:14.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:14.470 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:05:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:05:14.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:14.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:15.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:15.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:15.759 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:05:15.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:05:15.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:15.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:16.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:17.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:17.085 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:05:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:05:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:17.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:17.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:18.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:18.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:18.614 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:05:18.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:05:18.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:18.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:19.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:20.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:05:20.084 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:05:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:20.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:20.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:21.769 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:05:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:05:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:22.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:23.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:23.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:23.564 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:05:23.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:05:23.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:23.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:24.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:25.291 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:05:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:05:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:25.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:26.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:26.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:26.718 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:05:26.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:05:26.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:26.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:27.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:28.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:28.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:28.009 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:05:28.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:05:28.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:28.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:05:28.296 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2413333312,v1:127.0.0.1:6827/2413333312] [v2:127.0.0.1:6828/2413333312,v1:127.0.0.1:6829/2413333312] exists,up 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:05:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:05:28.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:05:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8654751d-2e7e-4f32-989a-59be39897f3e 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 8654751d-2e7e-4f32-989a-59be39897f3e' 2024-09-10T21:05:28.304 INFO:tasks.workunit.client.0.smithi181.stdout:add osd4 8654751d-2e7e-4f32-989a-59be39897f3e 2024-09-10T21:05:28.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:05:28.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYtOBmz5DFEhAAR/v/3xQGoAPrRdkVo448oQ== 2024-09-10T21:05:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYtOBmz5DFEhAAR/v/3xQGoAPrRdkVo448oQ=="}' 2024-09-10T21:05:28.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8654751d-2e7e-4f32-989a-59be39897f3e -i td/osd-scrub-repair/4/new.json 2024-09-10T21:05:28.651 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:05:28.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-10T21:05:28.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYtOBmz5DFEhAAR/v/3xQGoAPrRdkVo448oQ== --osd-uuid 8654751d-2e7e-4f32-989a-59be39897f3e 2024-09-10T21:05:28.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:28.688+0000 7f653f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:28.736 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:28.731+0000 7f653f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:28.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:28.732+0000 7f653f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:28.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:28.733+0000 7f653f5a8640 -1 bdev(0x55e56f7a5c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-10T21:05:28.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:28.733+0000 7f653f5a8640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-10T21:05:35.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-10T21:05:35.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:05:35.252 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd4 key to auth repository 2024-09-10T21:05:35.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T21:05:35.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:05:35.632 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.4 2024-09-10T21:05:35.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T21:05:35.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:05:35.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:05:35.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:05:35.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:05:35.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:35.656+0000 7f7f5d812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:35.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:35.658+0000 7f7f5d812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:35.663 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:35.659+0000 7f7f5d812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:05:35.910 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:05:35.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:35.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:36.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:37.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:37.024+0000 7f7f5d812640 -1 Falling back to public interface 2024-09-10T21:05:37.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:37.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:37.187 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:05:37.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:05:37.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:37.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:37.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:38.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:38.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:38.474 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:05:38.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:05:38.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:38.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:38.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:38.926 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:05:38.923+0000 7f7f5d812640 -1 osd.4 0 log_to_monitors true 2024-09-10T21:05:39.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:39.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:39.756 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:05:39.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:05:39.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:39.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:40.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:41.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:41.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:41.109 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:05:41.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:05:41.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:41.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:41.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:42.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:42.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:42.400 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:05:42.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:05:42.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:42.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:42.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:43.690 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:05:43.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:43.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:05:43.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:43.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:44.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:45.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:45.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:45.080 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:05:45.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:05:45.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:45.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:46.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:46.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:46.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:05:46.370 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:05:46.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:46.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:46.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:47.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:47.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:47.659 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:05:47.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:05:47.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:47.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:48.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:05:49.068 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:05:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:49.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:50.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:50.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:50.359 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:05:50.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:05:50.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:50.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:50.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:51.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:51.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:51.646 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:05:51.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:05:51.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:51.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:52.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:53.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:53.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:53.124 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:05:53.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:05:53.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:53.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:53.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:54.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:54.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:54.580 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:05:54.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:05:54.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:54.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:55.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:55.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:55.877 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:05:55.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:05:55.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:56.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:57.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:57.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:57.530 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:05:57.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:05:57.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:57.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:58.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:05:59.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:05:59.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:05:59.247 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:05:59.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:05:59.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:05:59.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:05:59.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:00.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:00.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:00.689 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:06:00.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:06:00.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:00.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:06:01.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:02.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:02.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:02.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:06:02.275 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:06:02.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:02.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:06:02.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:03.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:03.628 INFO:tasks.workunit.client.0.smithi181.stdout:20 2024-09-10T21:06:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-10T21:06:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:03.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T21:06:03.914 INFO:tasks.workunit.client.0.smithi181.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/725526154,v1:127.0.0.1:6835/725526154] [v2:127.0.0.1:6836/725526154,v1:127.0.0.1:6837/725526154] exists,up 8654751d-2e7e-4f32-989a-59be39897f3e 2024-09-10T21:06:03.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:06:03.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:06:03.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:06:03.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:06:03.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:06:03.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:06:03.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:06:03.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 aed94797-5266-4012-90c9-80165a1793b3' 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stdout:add osd5 aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:06:03.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:06:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC7tOBmrKWSNxAAEpt6xYVgcIH5eQBcsUan2g== 2024-09-10T21:06:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC7tOBmrKWSNxAAEpt6xYVgcIH5eQBcsUan2g=="}' 2024-09-10T21:06:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aed94797-5266-4012-90c9-80165a1793b3 -i td/osd-scrub-repair/5/new.json 2024-09-10T21:06:04.318 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:06:04.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-10T21:06:04.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7tOBmrKWSNxAAEpt6xYVgcIH5eQBcsUan2g== --osd-uuid aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:06:04.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:04.355+0000 7fe4926b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:04.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:04.394+0000 7fe4926b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:04.395+0000 7fe4926b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:04.395+0000 7fe4926b6640 -1 bdev(0x55adddd8bc00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-10T21:06:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:04.395+0000 7fe4926b6640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-10T21:06:10.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-10T21:06:10.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:06:10.776 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd5 key to auth repository 2024-09-10T21:06:10.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T21:06:10.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:06:11.135 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.5 2024-09-10T21:06:11.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T21:06:11.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:06:11.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:06:11.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:06:11.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:06:11.162 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:11.159+0000 7fbe16a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:11.164 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:11.161+0000 7fbe16a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:11.165 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:11.162+0000 7fbe16a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:11.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T21:06:11.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:06:11.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:06:11.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:06:11.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:06:11.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:11.441 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:06:11.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:06:11.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:11.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:11.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:12.719 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:06:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:06:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:12.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:12.782+0000 7fbe16a18640 -1 Falling back to public interface 2024-09-10T21:06:13.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:14.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:14.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:14.008 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:06:14.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:06:14.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:14.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:14.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:14.377 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:14.373+0000 7fbe16a18640 -1 osd.5 0 log_to_monitors true 2024-09-10T21:06:15.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:15.296 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:06:15.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:06:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:15.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:15.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:16.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:16.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:16.745 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:06:16.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:06:16.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:16.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:17.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:18.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:18.108 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:06:18.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:06:18.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:18.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:18.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:19.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:19.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:19.447 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:06:19.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:06:19.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:19.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:19.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:20.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:20.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:20.787 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:06:20.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:06:20.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:20.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:21.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:22.093 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:06:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:06:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:22.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:23.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:23.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:23.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:06:23.486 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:06:23.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:23.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:23.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:24.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:24.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:24.774 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:06:24.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:06:24.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:24.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:25.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:26.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:26.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:26.091 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:06:26.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:06:26.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:26.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:26.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:27.485 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:06:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:06:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:29.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:29.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:29.304 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:06:29.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:06:29.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:29.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:30.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:31.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:31.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:31.125 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:06:31.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:06:31.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:31.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:31.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:32.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:32.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:32.510 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:06:32.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:06:32.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:32.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:33.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:34.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:34.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:34.231 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:06:34.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:06:34.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:34.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:34.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:35.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:35.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:35.900 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:06:35.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:06:35.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:35.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:37.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:37.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:37.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:06:37.628 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:06:37.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:37.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:06:37.939 INFO:tasks.workunit.client.0.smithi181.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/898579715,v1:127.0.0.1:6843/898579715] [v2:127.0.0.1:6844/898579715,v1:127.0.0.1:6845/898579715] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T21:06:37.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:06:37.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:06:37.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=736c1ad7-5799-43c6-9af0-d8a306132477 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 736c1ad7-5799-43c6-9af0-d8a306132477' 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stdout:add osd6 736c1ad7-5799-43c6-9af0-d8a306132477 2024-09-10T21:06:37.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:06:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdtOBmXaoaORAA0CH9eaGLMCOVvf0zXJEVgg== 2024-09-10T21:06:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdtOBmXaoaORAA0CH9eaGLMCOVvf0zXJEVgg=="}' 2024-09-10T21:06:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 736c1ad7-5799-43c6-9af0-d8a306132477 -i td/osd-scrub-repair/6/new.json 2024-09-10T21:06:38.413 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:06:38.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-10T21:06:38.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdtOBmXaoaORAA0CH9eaGLMCOVvf0zXJEVgg== --osd-uuid 736c1ad7-5799-43c6-9af0-d8a306132477 2024-09-10T21:06:38.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:38.450+0000 7f8af8c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:38.506 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:38.502+0000 7f8af8c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:38.507 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:38.504+0000 7f8af8c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:38.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:38.504+0000 7f8af8c13640 -1 bdev(0x556acd9a0800 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-10T21:06:38.508 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:38.504+0000 7f8af8c13640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-10T21:06:44.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-10T21:06:44.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:06:44.605 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd6 key to auth repository 2024-09-10T21:06:44.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T21:06:44.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:06:44.967 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.6 2024-09-10T21:06:44.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T21:06:44.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:06:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:06:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:06:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:06:44.994 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:44.990+0000 7f4683c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:44.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:44.992+0000 7f4683c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:44.997 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:44.993+0000 7f4683c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:45.244 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:06:45.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:06:45.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:45.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:45.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:46.536 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:06:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:06:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:46.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:46.630 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:46.626+0000 7f4683c1b640 -1 Falling back to public interface 2024-09-10T21:06:46.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:47.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:47.826 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:06:47.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:06:47.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:47.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:48.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:48.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:06:48.541+0000 7f4683c1b640 -1 osd.6 0 log_to_monitors true 2024-09-10T21:06:49.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:49.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:49.117 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:06:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:06:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:49.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:49.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:50.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:50.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:50.694 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:06:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:06:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:50.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:51.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:51.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:51.986 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:06:51.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:06:51.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:51.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:52.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:53.353 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:06:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:06:53.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:53.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:53.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:54.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:54.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:54.660 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:06:54.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:06:54.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:54.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:54.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:55.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:55.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:55.947 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:06:55.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:06:55.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:55.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:57.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:57.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:57.359 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:06:57.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:06:57.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:57.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:57.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:06:58.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:06:58.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:06:58.774 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:06:58.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:06:58.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:06:58.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:06:59.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:00.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:00.063 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:07:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:07:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:00.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:01.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:01.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:01.349 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:07:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:07:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:03.055 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:07:03.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:07:03.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:03.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:04.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:05.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:05.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:05.117 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:07:05.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:07:05.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:05.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:05.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:06.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:06.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:06.878 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:07:06.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:07:06.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:06.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:07.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:08.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:08.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:08.406 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:07:08.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:07:08.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:08.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:08.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:09.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:09.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:09.825 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:07:09.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:07:09.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:09.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:10.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:11.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:11.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:11.606 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:07:11.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:07:11.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:11.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:07:12.925 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:07:12.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:12.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T21:07:13.211 INFO:tasks.workunit.client.0.smithi181.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3154822394,v1:127.0.0.1:6851/3154822394] [v2:127.0.0.1:6852/3154822394,v1:127.0.0.1:6853/3154822394] exists,up 736c1ad7-5799-43c6-9af0-d8a306132477 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:07:13.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:07:13.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:07:13.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5' 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stdout:add osd7 c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5 2024-09-10T21:07:13.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:07:13.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQABteBmYIyzDRAAtt49EpyozHibcGun1AepGw== 2024-09-10T21:07:13.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQABteBmYIyzDRAAtt49EpyozHibcGun1AepGw=="}' 2024-09-10T21:07:13.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5 -i td/osd-scrub-repair/7/new.json 2024-09-10T21:07:13.600 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:07:13.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-10T21:07:13.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABteBmYIyzDRAAtt49EpyozHibcGun1AepGw== --osd-uuid c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5 2024-09-10T21:07:13.642 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:13.638+0000 7f9c11fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:13.652 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:13.648+0000 7f9c11fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:13.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:13.649+0000 7f9c11fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:13.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:13.649+0000 7f9c11fb2640 -1 bdev(0x5568b30dbc00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-10T21:07:13.653 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:13.649+0000 7f9c11fb2640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-10T21:07:19.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-10T21:07:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:07:19.594 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd7 key to auth repository 2024-09-10T21:07:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-10T21:07:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:07:19.951 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.7 2024-09-10T21:07:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-10T21:07:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:07:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:07:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:07:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:07:19.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:19.975+0000 7fef96c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:19.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:19.977+0000 7fef96c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:19.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:19.978+0000 7fef96c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:07:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:20.230 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:07:20.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:07:20.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:20.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:20.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:21.101 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:21.097+0000 7fef96c0b640 -1 Falling back to public interface 2024-09-10T21:07:21.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:21.507 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:07:21.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:07:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:21.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:21.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:22.680 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:22.676+0000 7fef96c0b640 -1 osd.7 0 log_to_monitors true 2024-09-10T21:07:22.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:22.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:22.795 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:07:22.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:07:22.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:22.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:23.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:24.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:24.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:24.158 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:07:24.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:07:24.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:24.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:24.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:25.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:25.466 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:07:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:07:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:25.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:25.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:26.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:26.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:26.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:07:26.768 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:07:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:27.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:28.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:28.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:28.059 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:07:28.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:07:28.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:28.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:28.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:29.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:29.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:29.461 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:07:29.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:07:29.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:29.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:29.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:30.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:30.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:30.851 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:07:30.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:07:30.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:30.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:31.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:32.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:32.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:32.141 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:07:32.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:07:32.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:32.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:32.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:33.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:33.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:33.480 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:07:33.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:07:33.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:33.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:33.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:34.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:34.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:34.895 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:07:34.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:07:34.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:34.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:35.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:36.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:36.187 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:07:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:07:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:36.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:36.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:37.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:37.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:37.509 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:07:37.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:07:37.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:37.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:38.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:39.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:39.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:39.221 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:07:39.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:07:39.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:39.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:39.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:40.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:40.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:40.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:07:40.759 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:07:40.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:40.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:41.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:42.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:42.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:42.427 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:07:42.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:07:42.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:42.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:43.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:44.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:44.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:44.013 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:07:44.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:07:44.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:44.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:44.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:45.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:45.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:45.629 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:07:45.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:07:45.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:45.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T21:07:45.981 INFO:tasks.workunit.client.0.smithi181.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4031690983,v1:127.0.0.1:6859/4031690983] [v2:127.0.0.1:6860/4031690983,v1:127.0.0.1:6861/4031690983] exists,up c8d86f6b-0a7b-4bb0-bdca-0b1d78dd29a5 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:07:45.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:07:45.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:07:45.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:07:45.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b3dda7e4-c8fa-4386-8cad-e66519b69a9b 2024-09-10T21:07:45.990 INFO:tasks.workunit.client.0.smithi181.stdout:add osd8 b3dda7e4-c8fa-4386-8cad-e66519b69a9b 2024-09-10T21:07:45.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 b3dda7e4-c8fa-4386-8cad-e66519b69a9b' 2024-09-10T21:07:45.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:07:46.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiteBmSi8CABAAUY9KtK/CKiXxWKbTpNjrgg== 2024-09-10T21:07:46.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiteBmSi8CABAAUY9KtK/CKiXxWKbTpNjrgg=="}' 2024-09-10T21:07:46.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b3dda7e4-c8fa-4386-8cad-e66519b69a9b -i td/osd-scrub-repair/8/new.json 2024-09-10T21:07:46.350 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:07:46.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-10T21:07:46.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiteBmSi8CABAAUY9KtK/CKiXxWKbTpNjrgg== --osd-uuid b3dda7e4-c8fa-4386-8cad-e66519b69a9b 2024-09-10T21:07:46.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:46.388+0000 7fb720018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:46.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:46.431+0000 7fb720018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:46.436 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:46.432+0000 7fb720018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:46.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:46.432+0000 7fb720018640 -1 bdev(0x562c5f3b3c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-10T21:07:46.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:46.432+0000 7fb720018640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-10T21:07:52.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-10T21:07:52.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:07:52.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-10T21:07:52.854 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd8 key to auth repository 2024-09-10T21:07:52.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:07:53.214 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.8 2024-09-10T21:07:53.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-10T21:07:53.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:07:53.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:07:53.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:07:53.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:07:53.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:53.238+0000 7fe06dc87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:53.240+0000 7fe06dc87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:53.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:53.241+0000 7fe06dc87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:07:53.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-10T21:07:53.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:07:53.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:53.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:07:53.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:54.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:54.602+0000 7fe06dc87640 -1 Falling back to public interface 2024-09-10T21:07:54.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:54.793 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:07:54.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:54.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:07:54.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:54.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:07:55.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:56.082 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:07:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:07:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:56.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:07:56.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:07:56.206+0000 7fe06dc87640 -1 osd.8 0 log_to_monitors true 2024-09-10T21:07:56.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:57.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:57.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:57.420 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:07:57.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:07:57.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:57.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:07:57.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:07:58.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:07:58.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:07:58.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:07:58.804 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:07:58.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:07:58.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:07:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:00.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:00.117 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:08:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:08:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:00.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:00.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:01.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:01.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:01.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:08:01.526 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:08:01.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:01.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:01.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:02.934 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:08:02.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:08:02.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:02.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:03.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:04.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:04.246 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:08:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:08:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:04.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:04.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:05.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:05.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:05.538 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:08:05.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:08:05.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:05.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:05.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:06.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:06.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:06.946 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:08:06.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:08:06.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:06.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:08.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:08.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:08.279 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:08:08.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:08:08.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:08.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:08.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:09.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:09.598 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:08:09.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:09.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:08:09.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:09.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:09.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:10.952 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:08:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:08:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:10.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:11.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:12.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:12.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:12.397 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:08:12.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:08:12.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:12.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:12.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:13.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:13.978 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:08:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:08:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:13.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:14.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:15.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:15.386 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:08:15.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:08:15.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:15.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:15.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:16.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:16.965 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:08:16.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:16.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:08:16.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:16.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:17.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:18.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:18.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:18.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:08:18.544 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:08:18.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:18.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:19.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:20.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:20.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:20.209 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:08:20.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:08:20.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:20.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T21:08:20.594 INFO:tasks.workunit.client.0.smithi181.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3801533917,v1:127.0.0.1:6867/3801533917] [v2:127.0.0.1:6868/3801533917,v1:127.0.0.1:6869/3801533917] exists,up b3dda7e4-c8fa-4386-8cad-e66519b69a9b 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-10T21:08:20.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:08:20.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:08:20.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 4022b805-5ff0-4b04-ac5e-8f8ae621adfd' 2024-09-10T21:08:20.601 INFO:tasks.workunit.client.0.smithi181.stdout:add osd9 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:08:20.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:08:20.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEteBmxSKHJBAAxcuePZsCjxvaHny41Z6+Ig== 2024-09-10T21:08:20.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEteBmxSKHJBAAxcuePZsCjxvaHny41Z6+Ig=="}' 2024-09-10T21:08:20.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4022b805-5ff0-4b04-ac5e-8f8ae621adfd -i td/osd-scrub-repair/9/new.json 2024-09-10T21:08:20.941 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:08:20.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-10T21:08:20.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEteBmxSKHJBAAxcuePZsCjxvaHny41Z6+Ig== --osd-uuid 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:08:20.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:20.979+0000 7f9cf261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:21.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:21.022+0000 7f9cf261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:21.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:21.024+0000 7f9cf261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:21.024+0000 7f9cf261e640 -1 bdev(0x562afaa01c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-10T21:08:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:21.024+0000 7f9cf261e640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-10T21:08:27.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-10T21:08:27.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:08:27.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-10T21:08:27.699 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd9 key to auth repository 2024-09-10T21:08:27.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:08:28.061 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.9 2024-09-10T21:08:28.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-10T21:08:28.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:08:28.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:08:28.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:08:28.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:08:28.088 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:28.084+0000 7f2cac211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:28.090 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:28.086+0000 7f2cac211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:28.091 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:28.087+0000 7f2cac211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:08:28.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:08:28.340 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:08:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:28.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:28.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:29.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:29.462+0000 7f2cac211640 -1 Falling back to public interface 2024-09-10T21:08:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:08:29.620 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:08:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:29.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:30.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:30.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:30.909 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:08:30.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:08:30.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:30.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:31.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:08:31.049+0000 7f2cac211640 -1 osd.9 0 log_to_monitors true 2024-09-10T21:08:31.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:32.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:32.253 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:08:32.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:32.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:08:32.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:32.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:32.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:33.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:33.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:33.541 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:08:33.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:08:33.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:33.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:33.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:34.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:34.859 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:08:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:08:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:35.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:36.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:36.146 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:08:36.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:36.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:08:36.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:36.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:36.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:37.436 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:08:37.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:08:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:37.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:37.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:38.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:38.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:38.781 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:08:38.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:08:38.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:38.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:39.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:40.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:40.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:40.068 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:08:40.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:08:40.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:40.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:40.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:41.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:41.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:41.360 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:08:41.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:08:41.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:41.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:41.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:42.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:42.719 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:08:42.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:42.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:08:42.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:42.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:43.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:44.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:44.032 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:08:44.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:44.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:08:44.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:44.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:44.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:45.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:45.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:45.504 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:08:45.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:08:45.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:45.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:46.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:47.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:47.289 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:08:47.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:47.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:08:47.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:47.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:47.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:48.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:48.780 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:08:48.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:48.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:08:48.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:48.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:49.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:50.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:50.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:50.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:08:50.286 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:08:50.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:50.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:50.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:51.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:51.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:51.666 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:08:51.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:08:51.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:51.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:52.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:53.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:53.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:53.272 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:08:53.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:08:53.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:53.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:53.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:08:54.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:08:54.981 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:08:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:08:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:08:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:08:54.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:08:55.330 INFO:tasks.workunit.client.0.smithi181.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2696597228,v1:127.0.0.1:6875/2696597228] [v2:127.0.0.1:6876/2696597228,v1:127.0.0.1:6877/2696597228] exists,up 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:08:55.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:08:55.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:08:55.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:08:55.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-10T21:08:55.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:08:55.613 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:08:55.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:08:55.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:08:56.029 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:08:56.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:08:57.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:09:00.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-10T21:09:00.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:09:00.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:09:00.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:09:00.402 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:09:00.403 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:09:00.403 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:09:00.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:09:00.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:09:00.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:09:00.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:09:00.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:09:00.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:09:00.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:09:00.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:09:00.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:09:00.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:09:00.785 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:00.785 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:00.785 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:00.785 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:00.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:09:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836546 2024-09-10T21:09:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836546 2024-09-10T21:09:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546' 2024-09-10T21:09:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:00.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:09:00.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673020 2024-09-10T21:09:00.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673020 2024-09-10T21:09:00.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020' 2024-09-10T21:09:00.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:00.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:09:01.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509496 2024-09-10T21:09:01.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509496 2024-09-10T21:09:01.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496' 2024-09-10T21:09:01.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:09:01.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345967 2024-09-10T21:09:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345967 2024-09-10T21:09:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967' 2024-09-10T21:09:01.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:09:01.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182442 2024-09-10T21:09:01.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182442 2024-09-10T21:09:01.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442' 2024-09-10T21:09:01.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.352 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:09:01.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018915 2024-09-10T21:09:01.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018915 2024-09-10T21:09:01.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442 5-128849018915' 2024-09-10T21:09:01.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:09:01.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855389 2024-09-10T21:09:01.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855389 2024-09-10T21:09:01.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442 5-128849018915 6-150323855389' 2024-09-10T21:09:01.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.507 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:09:01.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691859 2024-09-10T21:09:01.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691859 2024-09-10T21:09:01.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442 5-128849018915 6-150323855389 7-171798691859' 2024-09-10T21:09:01.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:09:01.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528334 2024-09-10T21:09:01.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528334 2024-09-10T21:09:01.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442 5-128849018915 6-150323855389 7-171798691859 8-193273528334' 2024-09-10T21:09:01.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:01.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:09:01.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364809 2024-09-10T21:09:01.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364809 2024-09-10T21:09:01.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345967 4-107374182442 5-128849018915 6-150323855389 7-171798691859 8-193273528334 9-214748364809' 2024-09-10T21:09:01.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:01.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836546 2024-09-10T21:09:01.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:01.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:09:01.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836546 2024-09-10T21:09:01.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:01.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836546 2024-09-10T21:09:01.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-09-10T21:09:01.901 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836546 2024-09-10T21:09:01.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:09:02.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836546 -lt 21474836546 2024-09-10T21:09:02.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:02.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673020 2024-09-10T21:09:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:02.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:09:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673020 2024-09-10T21:09:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:02.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673020 2024-09-10T21:09:02.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-09-10T21:09:02.180 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673020 2024-09-10T21:09:02.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673020 -lt 42949673020 2024-09-10T21:09:02.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:02.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509496 2024-09-10T21:09:02.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:09:02.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509496 2024-09-10T21:09:02.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:02.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509496 2024-09-10T21:09:02.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-09-10T21:09:02.460 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509496 2024-09-10T21:09:02.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:09:02.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509496 -lt 64424509496 2024-09-10T21:09:02.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:02.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345967 2024-09-10T21:09:02.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:02.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:09:02.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345967 2024-09-10T21:09:02.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:02.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345967 2024-09-10T21:09:02.739 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345967 2024-09-10T21:09:02.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-09-10T21:09:02.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:09:03.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345967 -lt 85899345967 2024-09-10T21:09:03.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:03.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182442 2024-09-10T21:09:03.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:03.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:09:03.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182442 2024-09-10T21:09:03.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:03.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182442 2024-09-10T21:09:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-09-10T21:09:03.020 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.4 seq 107374182442 2024-09-10T21:09:03.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:09:03.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182441 -lt 107374182442 2024-09-10T21:09:03.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:09:04.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:09:04.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:09:04.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182442 -lt 107374182442 2024-09-10T21:09:04.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:04.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018915 2024-09-10T21:09:04.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:04.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:09:04.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018915 2024-09-10T21:09:04.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:04.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018915 2024-09-10T21:09:04.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-09-10T21:09:04.614 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.5 seq 128849018915 2024-09-10T21:09:04.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:09:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018916 -lt 128849018915 2024-09-10T21:09:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:04.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855389 2024-09-10T21:09:04.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:04.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:09:04.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855389 2024-09-10T21:09:04.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:04.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855389 2024-09-10T21:09:04.895 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.6 seq 150323855389 2024-09-10T21:09:04.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-09-10T21:09:04.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:09:05.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855390 -lt 150323855389 2024-09-10T21:09:05.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:05.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691859 2024-09-10T21:09:05.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:05.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:09:05.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691859 2024-09-10T21:09:05.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691859 2024-09-10T21:09:05.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691859' 2024-09-10T21:09:05.175 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.7 seq 171798691859 2024-09-10T21:09:05.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:09:05.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691859 -lt 171798691859 2024-09-10T21:09:05.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:05.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528334 2024-09-10T21:09:05.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:05.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:09:05.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528334 2024-09-10T21:09:05.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528334 2024-09-10T21:09:05.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528334' 2024-09-10T21:09:05.457 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.8 seq 193273528334 2024-09-10T21:09:05.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:09:05.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528334 -lt 193273528334 2024-09-10T21:09:05.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:05.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364809 2024-09-10T21:09:05.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:05.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:09:05.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364809 2024-09-10T21:09:05.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:05.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364809 2024-09-10T21:09:05.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-09-10T21:09:05.741 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.9 seq 214748364809 2024-09-10T21:09:05.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:09:06.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364809 -lt 214748364809 2024-09-10T21:09:06.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:09:06.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:06.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:06.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:09:06.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:09:06.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:09:06.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:09:06.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:09:06.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:09:06.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:09:06.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:09:06.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:09:06.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:09:06.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:06.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:06.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:09:06.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:09:06.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-10T21:09:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:09:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:09:08.091 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:09:08.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:09:09.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T21:09:09.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T21:09:09.574 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 2 allow_ec_overwrites to true 2024-09-10T21:09:09.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:09:09.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:09:09.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:09:09.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:09:09.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:09:09.586 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:09:09.586 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:09:09.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:09:09.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:09:09.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:09:09.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:09:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:09.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:09:10.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836549 2024-09-10T21:09:10.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836549 2024-09-10T21:09:10.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549' 2024-09-10T21:09:10.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:09:10.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-10T21:09:10.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-10T21:09:10.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023' 2024-09-10T21:09:10.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:09:10.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509499 2024-09-10T21:09:10.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509499 2024-09-10T21:09:10.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499' 2024-09-10T21:09:10.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:09:10.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345970 2024-09-10T21:09:10.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345970 2024-09-10T21:09:10.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970' 2024-09-10T21:09:10.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:09:10.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182445 2024-09-10T21:09:10.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182445 2024-09-10T21:09:10.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445' 2024-09-10T21:09:10.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:09:10.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018918 2024-09-10T21:09:10.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018918 2024-09-10T21:09:10.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445 5-128849018918' 2024-09-10T21:09:10.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:09:10.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855392 2024-09-10T21:09:10.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855392 2024-09-10T21:09:10.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445 5-128849018918 6-150323855392' 2024-09-10T21:09:10.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:10.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:09:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691861 2024-09-10T21:09:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691861 2024-09-10T21:09:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445 5-128849018918 6-150323855392 7-171798691861' 2024-09-10T21:09:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:11.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:09:11.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528337 2024-09-10T21:09:11.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528337 2024-09-10T21:09:11.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445 5-128849018918 6-150323855392 7-171798691861 8-193273528337' 2024-09-10T21:09:11.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:11.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:09:11.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364812 2024-09-10T21:09:11.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364812 2024-09-10T21:09:11.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509499 3-85899345970 4-107374182445 5-128849018918 6-150323855392 7-171798691861 8-193273528337 9-214748364812' 2024-09-10T21:09:11.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:11.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836549 2024-09-10T21:09:11.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:11.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:09:11.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836549 2024-09-10T21:09:11.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:11.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836549 2024-09-10T21:09:11.259 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836549 2024-09-10T21:09:11.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-09-10T21:09:11.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:09:11.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836549 2024-09-10T21:09:11.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:11.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-10T21:09:11.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:11.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:09:11.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-10T21:09:11.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:11.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-10T21:09:11.543 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673023 2024-09-10T21:09:11.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-10T21:09:11.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:11.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-10T21:09:11.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:11.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509499 2024-09-10T21:09:11.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:11.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:09:11.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509499 2024-09-10T21:09:11.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:11.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509499 2024-09-10T21:09:11.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-09-10T21:09:11.849 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509499 2024-09-10T21:09:11.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:09:12.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509499 -lt 64424509499 2024-09-10T21:09:12.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:12.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345970 2024-09-10T21:09:12.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:12.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:09:12.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345970 2024-09-10T21:09:12.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:12.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345970 2024-09-10T21:09:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-09-10T21:09:12.130 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345970 2024-09-10T21:09:12.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:09:12.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345970 -lt 85899345970 2024-09-10T21:09:12.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:12.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182445 2024-09-10T21:09:12.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:12.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:09:12.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182445 2024-09-10T21:09:12.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:12.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182445 2024-09-10T21:09:12.409 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.4 seq 107374182445 2024-09-10T21:09:12.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-09-10T21:09:12.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:09:12.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182445 -lt 107374182445 2024-09-10T21:09:12.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:12.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018918 2024-09-10T21:09:12.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:12.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:09:12.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018918 2024-09-10T21:09:12.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:12.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018918 2024-09-10T21:09:12.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-09-10T21:09:12.695 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.5 seq 128849018918 2024-09-10T21:09:12.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:09:12.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018918 -lt 128849018918 2024-09-10T21:09:12.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:12.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855392 2024-09-10T21:09:12.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:12.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:09:12.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855392 2024-09-10T21:09:12.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:12.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855392 2024-09-10T21:09:12.976 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.6 seq 150323855392 2024-09-10T21:09:12.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-09-10T21:09:12.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:09:13.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855392 -lt 150323855392 2024-09-10T21:09:13.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:13.256 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691861 2024-09-10T21:09:13.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:13.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:09:13.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691861 2024-09-10T21:09:13.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:13.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691861 2024-09-10T21:09:13.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-09-10T21:09:13.260 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.7 seq 171798691861 2024-09-10T21:09:13.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:09:13.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691862 -lt 171798691861 2024-09-10T21:09:13.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:13.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528337 2024-09-10T21:09:13.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:13.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:09:13.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528337 2024-09-10T21:09:13.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:13.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528337 2024-09-10T21:09:13.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-09-10T21:09:13.595 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.8 seq 193273528337 2024-09-10T21:09:13.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:09:13.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528337 -lt 193273528337 2024-09-10T21:09:13.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:13.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364812 2024-09-10T21:09:13.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:13.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:09:13.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364812 2024-09-10T21:09:13.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:13.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364812 2024-09-10T21:09:13.874 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.9 seq 214748364812 2024-09-10T21:09:13.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-09-10T21:09:13.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:09:14.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364812 -lt 214748364812 2024-09-10T21:09:14.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:09:14.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:14.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:09:14.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:09:14.507 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:09:14.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:09:14.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:09:14.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:14.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:15.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:09:15.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:09:15.626 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:09:15.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:09:16.120 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:09:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:09:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:09:16.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:09:16.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T21:09:16.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T21:09:16.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:09:16.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:09:16.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:09:17.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-10T21:09:17.064 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T21:09:17.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:09:17.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T21:09:17.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-10T21:09:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:09:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:9 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr:7' 2024-09-10T21:09:17.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:09:17.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:09:17.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:09:17.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:09:17.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:09:17.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:09:19.671 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:09:20.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:20.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:09:20.194 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.3 2024-09-10T21:09:20.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:09:20.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:09:20.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:09:20.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:09:20.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:09:20.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:09:20.223 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:20.219+0000 7f6784c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:20.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:20.222+0000 7f6784c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:20.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:20.223+0000 7f6784c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:09:20.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:09:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:09:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:20.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:09:20.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:21.573 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:21.570+0000 7f6784c11640 -1 Falling back to public interface 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:21.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:09:22.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:23.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:23.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:23.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:09:23.040 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:23.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:23.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:09:23.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:23.151+0000 7f6784c11640 -1 osd.3 69 log_to_monitors true 2024-09-10T21:09:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:09:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/3912325340,v1:127.0.0.1:6827/3912325340] [v2:127.0.0.1:6828/3912325340,v1:127.0.0.1:6829/3912325340] exists,up 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:09:24.677 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:09:24.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:09:24.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:09:24.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:09:24.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:09:25.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:09:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-10T21:09:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-10T21:09:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-10T21:09:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:09:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673027 2024-09-10T21:09:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673027 2024-09-10T21:09:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027' 2024-09-10T21:09:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:09:25.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-10T21:09:25.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-10T21:09:25.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503' 2024-09-10T21:09:25.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:09:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-10T21:09:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-10T21:09:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610' 2024-09-10T21:09:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:09:25.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182449 2024-09-10T21:09:25.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182449 2024-09-10T21:09:25.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449' 2024-09-10T21:09:25.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:09:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018922 2024-09-10T21:09:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018922 2024-09-10T21:09:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449 5-128849018922' 2024-09-10T21:09:25.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:09:25.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855396 2024-09-10T21:09:25.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855396 2024-09-10T21:09:25.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449 5-128849018922 6-150323855396' 2024-09-10T21:09:25.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:09:25.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-10T21:09:25.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-10T21:09:25.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449 5-128849018922 6-150323855396 7-171798691865' 2024-09-10T21:09:25.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:25.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:09:26.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528341 2024-09-10T21:09:26.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528341 2024-09-10T21:09:26.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449 5-128849018922 6-150323855396 7-171798691865 8-193273528341' 2024-09-10T21:09:26.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:26.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:09:26.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364816 2024-09-10T21:09:26.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364816 2024-09-10T21:09:26.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-313532612610 4-107374182449 5-128849018922 6-150323855396 7-171798691865 8-193273528341 9-214748364816' 2024-09-10T21:09:26.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:26.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-10T21:09:26.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:26.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:09:26.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-10T21:09:26.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-10T21:09:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-10T21:09:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836553 2024-09-10T21:09:26.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:09:26.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836553 2024-09-10T21:09:26.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:26.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673027 2024-09-10T21:09:26.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:26.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:09:26.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673027 2024-09-10T21:09:26.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:26.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673027 2024-09-10T21:09:26.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-09-10T21:09:26.463 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673027 2024-09-10T21:09:26.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:26.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673027 2024-09-10T21:09:26.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:09:27.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:09:27.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:28.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673027 2024-09-10T21:09:28.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:28.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-10T21:09:28.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:28.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:09:28.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-10T21:09:28.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:28.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-10T21:09:28.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-10T21:09:28.026 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509503 2024-09-10T21:09:28.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:09:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509503 2024-09-10T21:09:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:28.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612610 2024-09-10T21:09:28.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:28.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:09:28.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612610 2024-09-10T21:09:28.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:28.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-10T21:09:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-09-10T21:09:28.309 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 313532612610 2024-09-10T21:09:28.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:09:28.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-10T21:09:28.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:28.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182449 2024-09-10T21:09:28.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:28.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:09:28.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182449 2024-09-10T21:09:28.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:28.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182449 2024-09-10T21:09:28.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2024-09-10T21:09:28.586 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182449 2024-09-10T21:09:28.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:09:28.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182449 -lt 107374182449 2024-09-10T21:09:28.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:28.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018922 2024-09-10T21:09:28.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:28.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:09:28.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018922 2024-09-10T21:09:28.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:28.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018922 2024-09-10T21:09:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-09-10T21:09:28.868 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 128849018922 2024-09-10T21:09:28.868 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:09:29.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018922 -lt 128849018922 2024-09-10T21:09:29.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:29.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855396 2024-09-10T21:09:29.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:29.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:09:29.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855396 2024-09-10T21:09:29.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855396 2024-09-10T21:09:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-09-10T21:09:29.149 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855396 2024-09-10T21:09:29.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:09:29.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855397 -lt 150323855396 2024-09-10T21:09:29.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:29.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691865 2024-09-10T21:09:29.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:29.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:09:29.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691865 2024-09-10T21:09:29.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:29.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-10T21:09:29.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-09-10T21:09:29.473 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691865 2024-09-10T21:09:29.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:09:29.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691866 -lt 171798691865 2024-09-10T21:09:29.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:29.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528341 2024-09-10T21:09:29.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:29.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:09:29.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528341 2024-09-10T21:09:29.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:29.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528341 2024-09-10T21:09:29.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-09-10T21:09:29.751 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528341 2024-09-10T21:09:29.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:09:30.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528341 -lt 193273528341 2024-09-10T21:09:30.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:30.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364816 2024-09-10T21:09:30.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:30.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:09:30.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364816 2024-09-10T21:09:30.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:30.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364816 2024-09-10T21:09:30.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-09-10T21:09:30.073 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 214748364816 2024-09-10T21:09:30.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:09:30.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364816 -lt 214748364816 2024-09-10T21:09:30.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:09:30.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:30.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:09:30.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:09:30.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:09:30.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:09:30.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:30.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:09:31.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:09:31.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:09:31.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:31.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:09:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:09:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:09:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:09:32.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:09:32.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:09:32.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:08.028513+0000 '>' 2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:32.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:09:33.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:09:33.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:09:33.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:09:33.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:09:33.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:09:33.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:09:33.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:09:33.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:08.028513+0000 '>' 2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:33.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:09:34.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:09:34.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:09:34.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:09:34.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:09:34.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:09:34.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:09:34.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:09:34.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:08.028513+0000 '>' 2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:34.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:09:35.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:09:35.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:09:35.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:09:35.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:09:35.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:09:35.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:09:35.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:09:36.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:32.336913+0000 '>' 2024-09-10T21:09:08.028513+0000 2024-09-10T21:09:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:09:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:09:38.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:09:38.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:09:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:09:38.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:09:39.390 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:09:39.390 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T21:09:39.390 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:09:39.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:09:39.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:09:39.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:09:39.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:09:39.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:09:39.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:09:39.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:09:39.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:09:39.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:09:39.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:09:39.666 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.9 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:09:39.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:09:39.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:39.690+0000 7f1429e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:39.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:39.693+0000 7f1429e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:39.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:39.694+0000 7f1429e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:09:40.248 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:09:40.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:40.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:09:40.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:41.048+0000 7f1429e24640 -1 Falling back to public interface 2024-09-10T21:09:41.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:41.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:41.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:09:41.527 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:41.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:41.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:09:41.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:42.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:09:43.037 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:43.033+0000 7f1429e24640 -1 osd.9 74 log_to_monitors true 2024-09-10T21:09:43.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:09:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:44.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:44.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:09:44.489 INFO:tasks.workunit.client.0.smithi181.stderr:osd.9 up in weight 1 up_from 79 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/2192486767,v1:127.0.0.1:6875/2192486767] [v2:127.0.0.1:6876/2192486767,v1:127.0.0.1:6877/2192486767] exists,up 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:09:44.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:09:44.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:09:44.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:09:44.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:09:44.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:09:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:09:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:09:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:09:44.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:09:44.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:09:45.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:09:45.038 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:45.038 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:45.038 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:45.038 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:45.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:09:45.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836558 2024-09-10T21:09:45.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836558 2024-09-10T21:09:45.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558' 2024-09-10T21:09:45.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:45.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:09:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673032 2024-09-10T21:09:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673032 2024-09-10T21:09:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032' 2024-09-10T21:09:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:45.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:09:45.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509508 2024-09-10T21:09:45.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509508 2024-09-10T21:09:45.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508' 2024-09-10T21:09:45.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:45.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:09:45.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612615 2024-09-10T21:09:45.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612615 2024-09-10T21:09:45.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615' 2024-09-10T21:09:45.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:45.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:09:46.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182454 2024-09-10T21:09:46.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182454 2024-09-10T21:09:46.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454' 2024-09-10T21:09:46.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:46.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:09:46.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018927 2024-09-10T21:09:46.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018927 2024-09-10T21:09:46.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454 5-128849018927' 2024-09-10T21:09:46.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:46.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:09:46.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855401 2024-09-10T21:09:46.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855401 2024-09-10T21:09:46.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454 5-128849018927 6-150323855401' 2024-09-10T21:09:46.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:46.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:09:46.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691871 2024-09-10T21:09:46.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691871 2024-09-10T21:09:46.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454 5-128849018927 6-150323855401 7-171798691871' 2024-09-10T21:09:46.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:46.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:09:46.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528346 2024-09-10T21:09:46.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528346 2024-09-10T21:09:46.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454 5-128849018927 6-150323855401 7-171798691871 8-193273528346' 2024-09-10T21:09:46.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:46.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673032 2-64424509508 3-313532612615 4-107374182454 5-128849018927 6-150323855401 7-171798691871 8-193273528346 9-339302416386' 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836558 2024-09-10T21:09:46.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:46.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:09:46.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836558 2024-09-10T21:09:46.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:46.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836558 2024-09-10T21:09:46.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-10T21:09:46.525 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836558 2024-09-10T21:09:46.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:09:46.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-10T21:09:46.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:46.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673032 2024-09-10T21:09:46.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:46.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:09:46.802 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673032 2024-09-10T21:09:46.802 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:46.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673032 2024-09-10T21:09:46.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-09-10T21:09:46.803 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673032 2024-09-10T21:09:46.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:47.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673032 2024-09-10T21:09:47.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:09:48.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:09:48.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:09:48.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673032 -lt 42949673032 2024-09-10T21:09:48.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:48.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509508 2024-09-10T21:09:48.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:48.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:09:48.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509508 2024-09-10T21:09:48.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:48.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509508 2024-09-10T21:09:48.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-09-10T21:09:48.378 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509508 2024-09-10T21:09:48.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:09:48.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509508 -lt 64424509508 2024-09-10T21:09:48.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:48.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612615 2024-09-10T21:09:48.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:48.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:09:48.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612615 2024-09-10T21:09:48.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:48.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612615 2024-09-10T21:09:48.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-09-10T21:09:48.663 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 313532612615 2024-09-10T21:09:48.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:09:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612615 -lt 313532612615 2024-09-10T21:09:48.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:48.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182454 2024-09-10T21:09:48.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:48.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:09:48.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182454 2024-09-10T21:09:48.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:48.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182454 2024-09-10T21:09:48.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-09-10T21:09:48.946 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182454 2024-09-10T21:09:48.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:09:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182454 -lt 107374182454 2024-09-10T21:09:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:49.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018927 2024-09-10T21:09:49.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:09:49.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018927 2024-09-10T21:09:49.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:49.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018927 2024-09-10T21:09:49.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-09-10T21:09:49.225 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 128849018927 2024-09-10T21:09:49.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:09:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018928 -lt 128849018927 2024-09-10T21:09:49.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:49.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855401 2024-09-10T21:09:49.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:49.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:09:49.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855401 2024-09-10T21:09:49.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855401 2024-09-10T21:09:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-09-10T21:09:49.544 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855401 2024-09-10T21:09:49.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:09:49.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855402 -lt 150323855401 2024-09-10T21:09:49.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:49.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691871 2024-09-10T21:09:49.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:49.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:09:49.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691871 2024-09-10T21:09:49.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:49.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691871 2024-09-10T21:09:49.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-09-10T21:09:49.832 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691871 2024-09-10T21:09:49.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:09:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691871 -lt 171798691871 2024-09-10T21:09:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:50.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528346 2024-09-10T21:09:50.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:50.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:09:50.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528346 2024-09-10T21:09:50.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:50.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528346 2024-09-10T21:09:50.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-09-10T21:09:50.112 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528346 2024-09-10T21:09:50.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:09:50.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528346 -lt 193273528346 2024-09-10T21:09:50.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:09:50.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-339302416386 2024-09-10T21:09:50.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:09:50.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:09:50.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-339302416386 2024-09-10T21:09:50.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:09:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-10T21:09:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 339302416386' 2024-09-10T21:09:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 339302416386 2024-09-10T21:09:50.391 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:09:50.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416387 -lt 339302416386 2024-09-10T21:09:50.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:09:50.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:50.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:51.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:09:51.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:09:51.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:09:51.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:09:51.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:09:51.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:09:51.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:09:51.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:09:51.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:09:51.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:09:51.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:09:51.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:09:51.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:09:51.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:09:51.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:09:51.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:09:51.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:09:51.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:09:51.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:09:51.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:09:51.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:09:54.312 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:09:54.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:09:54.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:09:54.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:09:54.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:09:54.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:09:54.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:09:54.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.5 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:09:54.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:09:54.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:09:54.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:09:54.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:54.856+0000 7f957261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:54.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:54.859+0000 7f957261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:54.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:54.861+0000 7f957261e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:09:55.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:55.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:09:55.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:55.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:55.978+0000 7f957261e640 -1 Falling back to public interface 2024-09-10T21:09:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:09:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:56.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:56.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:09:56.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:57.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:09:57.704+0000 7f957261e640 -1 osd.5 81 log_to_monitors true 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:57.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:09:58.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:09:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:09:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:09:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:09:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:09:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:osd.5 up in weight 1 up_from 86 up_thru 86 down_at 82 last_clean_interval [30,81) [v2:127.0.0.1:6842/2095122874,v1:127.0.0.1:6843/2095122874] [v2:127.0.0.1:6844/2095122874,v1:127.0.0.1:6845/2095122874] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:09:59.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:09:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:09:59.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:09:59.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:09:59.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:09:59.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:09:59.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:09:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:09:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:09:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:09:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:09:59.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:09:59.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:09:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:09:59.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836562 2024-09-10T21:10:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836562 2024-09-10T21:10:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562' 2024-09-10T21:10:00.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:00.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-10T21:10:00.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-10T21:10:00.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036' 2024-09-10T21:10:00.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-10T21:10:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-10T21:10:00.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512' 2024-09-10T21:10:00.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612619 2024-09-10T21:10:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612619 2024-09-10T21:10:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619' 2024-09-10T21:10:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:00.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182458 2024-09-10T21:10:00.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182458 2024-09-10T21:10:00.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458' 2024-09-10T21:10:00.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:00.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187458 2024-09-10T21:10:00.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187458 2024-09-10T21:10:00.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458 5-369367187458' 2024-09-10T21:10:00.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:00.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855405 2024-09-10T21:10:00.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855405 2024-09-10T21:10:00.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458 5-369367187458 6-150323855405' 2024-09-10T21:10:00.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:00.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691874 2024-09-10T21:10:00.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691874 2024-09-10T21:10:00.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458 5-369367187458 6-150323855405 7-171798691874' 2024-09-10T21:10:00.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:00.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528350 2024-09-10T21:10:00.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528350 2024-09-10T21:10:00.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458 5-369367187458 6-150323855405 7-171798691874 8-193273528350' 2024-09-10T21:10:00.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:00.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:01.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416390 2024-09-10T21:10:01.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416390 2024-09-10T21:10:01.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673036 2-64424509512 3-313532612619 4-107374182458 5-369367187458 6-150323855405 7-171798691874 8-193273528350 9-339302416390' 2024-09-10T21:10:01.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:01.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836562 2024-09-10T21:10:01.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:01.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:01.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836562 2024-09-10T21:10:01.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:01.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836562 2024-09-10T21:10:01.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-10T21:10:01.097 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836562 2024-09-10T21:10:01.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:01.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836562 2024-09-10T21:10:01.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:01.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-10T21:10:01.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:01.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:01.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-10T21:10:01.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:01.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-10T21:10:01.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-10T21:10:01.378 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673036 2024-09-10T21:10:01.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:10:01.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-10T21:10:01.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:01.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-10T21:10:01.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-10T21:10:01.660 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509512 2024-09-10T21:10:01.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:01.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-10T21:10:01.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:01.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612619 2024-09-10T21:10:01.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:01.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:01.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612619 2024-09-10T21:10:01.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:01.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612619 2024-09-10T21:10:01.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-09-10T21:10:01.942 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 313532612619 2024-09-10T21:10:01.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:02.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612619 -lt 313532612619 2024-09-10T21:10:02.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:02.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182458 2024-09-10T21:10:02.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:02.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:02.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182458 2024-09-10T21:10:02.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:02.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182458 2024-09-10T21:10:02.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-09-10T21:10:02.225 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182458 2024-09-10T21:10:02.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182458 -lt 107374182458 2024-09-10T21:10:02.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:02.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187458 2024-09-10T21:10:02.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:02.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:10:02.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187458 2024-09-10T21:10:02.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187458 2024-09-10T21:10:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187458' 2024-09-10T21:10:02.506 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 369367187458 2024-09-10T21:10:02.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:10:02.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187458 -lt 369367187458 2024-09-10T21:10:02.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855405 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855405 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855405 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855405 2024-09-10T21:10:02.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:10:03.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855405 -lt 150323855405 2024-09-10T21:10:03.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:03.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691874 2024-09-10T21:10:03.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:03.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:10:03.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691874 2024-09-10T21:10:03.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:03.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691874 2024-09-10T21:10:03.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-09-10T21:10:03.113 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691874 2024-09-10T21:10:03.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:10:03.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691875 -lt 171798691874 2024-09-10T21:10:03.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:03.391 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528350 2024-09-10T21:10:03.391 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:03.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:10:03.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528350 2024-09-10T21:10:03.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528350 2024-09-10T21:10:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-10T21:10:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528350 2024-09-10T21:10:03.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:10:03.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528350 -lt 193273528350 2024-09-10T21:10:03.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:03.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-339302416390 2024-09-10T21:10:03.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:03.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:10:03.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-339302416390 2024-09-10T21:10:03.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:03.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416390 2024-09-10T21:10:03.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 339302416390' 2024-09-10T21:10:03.680 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 339302416390 2024-09-10T21:10:03.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:10:03.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416390 -lt 339302416390 2024-09-10T21:10:03.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:10:03.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:03.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:04.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:10:04.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:04.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:04.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:10:04.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:10:04.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:10:04.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:10:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-10T21:10:05.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:05.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:10:05.743 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:10:05.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:05.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:10:05.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:05.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:05.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:32.336913+0000 '>' 2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:06.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:07.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:07.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:07.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:07.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:07.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:07.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:07.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:07.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:32.336913+0000 '>' 2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:07.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:08.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:08.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:08.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:08.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:08.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:08.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:08.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:08.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:09:32.336913+0000 '>' 2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:08.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:09.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:09.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:09.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:09.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:06.464927+0000 '>' 2024-09-10T21:09:32.336913+0000 2024-09-10T21:10:09.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:10:09.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:10:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:10:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:10:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:10:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:10:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:10:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:12.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:10:12.767 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:10:12.767 INFO:tasks.workunit.client.0.smithi181.stdout:hinfo_key 2024-09-10T21:10:12.767 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:10:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:10:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:10:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:10:13.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:10:13.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:10:13.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:10:13.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:10:13.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:10:13.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:10:13.048 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.9 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:10:13.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:10:13.076 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:10:13.072+0000 7f945474b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:13.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:10:13.236+0000 7f945474b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:13.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:10:13.237+0000 7f945474b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:10:13.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:13.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:13.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:14.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:10:14.585+0000 7f945474b640 -1 Falling back to public interface 2024-09-10T21:10:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:10:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:10:14.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:14.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:15.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:16.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:16.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:16.487 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:10:16.482+0000 7f945474b640 -1 osd.9 88 log_to_monitors true 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:17.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:17.732 INFO:tasks.workunit.client.0.smithi181.stderr:osd.9 up in weight 1 up_from 92 up_thru 0 down_at 89 last_clean_interval [79,88) [v2:127.0.0.1:6874/1843898283,v1:127.0.0.1:6875/1843898283] [v2:127.0.0.1:6876/1843898283,v1:127.0.0.1:6877/1843898283] exists,up 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:10:17.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:10:17.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:10:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:10:17.734 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:10:17.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:10:17.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:10:17.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:10:17.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:10:18.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:10:18.208 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:10:18.208 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:4 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:5 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:6 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:7 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:8 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:9' 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:18.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:18.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-10T21:10:18.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-10T21:10:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-10T21:10:18.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:18.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:18.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-10T21:10:18.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-10T21:10:18.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040' 2024-09-10T21:10:18.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:18.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:18.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509517 2024-09-10T21:10:18.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509517 2024-09-10T21:10:18.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517' 2024-09-10T21:10:18.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:18.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:18.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612624 2024-09-10T21:10:18.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612624 2024-09-10T21:10:18.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624' 2024-09-10T21:10:18.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:18.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182463 2024-09-10T21:10:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182463 2024-09-10T21:10:19.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463' 2024-09-10T21:10:19.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:19.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:19.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187463 2024-09-10T21:10:19.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187463 2024-09-10T21:10:19.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463 5-369367187463' 2024-09-10T21:10:19.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:19.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:19.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855410 2024-09-10T21:10:19.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855410 2024-09-10T21:10:19.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463 5-369367187463 6-150323855410' 2024-09-10T21:10:19.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:19.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:19.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-10T21:10:19.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-10T21:10:19.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463 5-369367187463 6-150323855410 7-171798691879' 2024-09-10T21:10:19.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:19.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:19.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528354 2024-09-10T21:10:19.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528354 2024-09-10T21:10:19.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463 5-369367187463 6-150323855410 7-171798691879 8-193273528354' 2024-09-10T21:10:19.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:19.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991234 2024-09-10T21:10:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991234 2024-09-10T21:10:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509517 3-313532612624 4-107374182463 5-369367187463 6-150323855410 7-171798691879 8-193273528354 9-395136991234' 2024-09-10T21:10:19.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:19.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-10T21:10:19.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:19.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:19.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-10T21:10:19.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-10T21:10:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-10T21:10:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836567 2024-09-10T21:10:19.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:19.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-10T21:10:19.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-10T21:10:19.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:19.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:19.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-10T21:10:19.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:19.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-10T21:10:19.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-10T21:10:19.843 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949673040 2024-09-10T21:10:19.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:10:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-10T21:10:20.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:20.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509517 2024-09-10T21:10:20.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:20.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509517 2024-09-10T21:10:20.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509517 2024-09-10T21:10:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-09-10T21:10:20.123 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509517 2024-09-10T21:10:20.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:20.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509517 2024-09-10T21:10:20.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:20.402 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612624 2024-09-10T21:10:20.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:20.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612624 2024-09-10T21:10:20.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:20.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612624 2024-09-10T21:10:20.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-09-10T21:10:20.405 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.3 seq 313532612624 2024-09-10T21:10:20.406 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:20.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612624 -lt 313532612624 2024-09-10T21:10:20.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:20.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182463 2024-09-10T21:10:20.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:20.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:20.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182463 2024-09-10T21:10:20.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:20.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182463 2024-09-10T21:10:20.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-10T21:10:20.685 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.4 seq 107374182463 2024-09-10T21:10:20.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:20.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182463 -lt 107374182463 2024-09-10T21:10:20.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:20.963 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-369367187463 2024-09-10T21:10:20.963 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:20.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:10:20.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-369367187463 2024-09-10T21:10:20.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:20.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187463 2024-09-10T21:10:20.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 369367187463' 2024-09-10T21:10:20.966 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.5 seq 369367187463 2024-09-10T21:10:20.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:10:21.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187463 -lt 369367187463 2024-09-10T21:10:21.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:21.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855410 2024-09-10T21:10:21.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:21.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:10:21.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855410 2024-09-10T21:10:21.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:21.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855410 2024-09-10T21:10:21.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-09-10T21:10:21.246 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.6 seq 150323855410 2024-09-10T21:10:21.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:10:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855410 -lt 150323855410 2024-09-10T21:10:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:21.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-10T21:10:21.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:21.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:10:21.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-10T21:10:21.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:21.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-10T21:10:21.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-10T21:10:21.542 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.7 seq 171798691879 2024-09-10T21:10:21.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:10:21.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-10T21:10:21.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:21.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528354 2024-09-10T21:10:21.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:21.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:10:21.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528354 2024-09-10T21:10:21.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:21.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528354 2024-09-10T21:10:21.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-10T21:10:21.820 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.8 seq 193273528354 2024-09-10T21:10:21.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:10:22.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528355 -lt 193273528354 2024-09-10T21:10:22.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:22.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-395136991234 2024-09-10T21:10:22.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:22.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:10:22.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-395136991234 2024-09-10T21:10:22.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:22.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991234 2024-09-10T21:10:22.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 395136991234' 2024-09-10T21:10:22.101 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.9 seq 395136991234 2024-09-10T21:10:22.101 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:10:22.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 395136991234 -lt 395136991234 2024-09-10T21:10:22.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:10:22.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:22.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:22.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:22.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:10:22.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:22.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:22.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:10:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T21:10:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:10:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T21:10:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:10:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:10:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:10:23.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:23.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:23.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:23.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:23.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:23.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:23.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:23.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:10:23.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:23.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:23.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:24.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:10:24.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:10:24.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:10:24.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:10:24.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:10:24.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-10T21:10:24.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T21:10:24.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 247989"' 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 247989' 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 247990"' 2024-09-10T21:10:24.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 247990' 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 247989 247990' 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 247989 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/247992: /' 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:10:24.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/247994: /' 2024-09-10T21:10:27.358 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.358 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:10:27.358 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:10:27.359 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:10:27.360 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:10:27.361 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:10:27.361 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:10:27.361 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: start osd.5 2024-09-10T21:10:27.363 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:2247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:10:27.484 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-10T21:10:27.485 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:10:27.486 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:27.487 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: start osd.9 2024-09-10T21:10:27.488 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:10:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2024-09-10T21:10:27.388+0000 7f7d49c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2024-09-10T21:10:27.408+0000 7f7d49c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2024-09-10T21:10:27.409+0000 7f7d49c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 0 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2024-09-10T21:10:28.994+0000 7f7d49c0e640 -1 Falling back to public interface 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 1 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2 2024-09-10T21:10:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2024-09-10T21:10:31.229+0000 7f7d49c0e640 -1 osd.5 93 log_to_monitors true 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 3 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247992: osd.5 up in weight 1 up_from 98 up_thru 86 down_at 94 last_c47994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:10:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2024-09-10T21:10:27.511+0000 7f98c3422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2024-09-10T21:10:27.513+0000 7f98c3422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2024-09-10T21:10:27.514+0000 7f98c3422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 0 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 1 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2024-09-10T21:10:29.125+0000 7f98c3422640 -1 Falling back to public interface 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.071 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2024-09-10T21:10:31.504+0000 7f98c3422640 -1 osd.9 93 log_to_monitors true 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 3 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:32.072 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperslean_interval [86,93) [v2:127.0.0.1:6842/1038780887,v1:127.0.0.1:6843/1038780887] [v2:127.0.0.1:6844/1038780887,v1:127.0.0.1:6845/1038780887] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:10:32.959 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:10:32.960 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 1 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 2 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 3 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 4 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 5 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 6 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 7 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 8 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: 9' 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836571 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836571 2024-09-10T21:10:32.961 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571' 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673044 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673044 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044' 2024-09-10T21:10:32.962 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.258 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 4 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: osd.9 up in weight 1 up_from 99 up_thru 0 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6874/1334120837,v1:127.0.0.1:6875/1334120837] [v2:127.0.0.1:6876/1334120837,v1:127.0.0.1:6877/1334120837] exists,up 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:10:34.259 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:10:34.260 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 1 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 2 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 3 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 4 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 5 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 6 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 7 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 8 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: 9' 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.261 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:34.262 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-10T21:10:34.262 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-10T21:10:34.262 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-10T21:10:34.262 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.578 INFO:tasks.workunit.client.0.smithi181.stderr:247994: ///standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:34.578 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-10T21:10:34.578 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-10T21:10:34.578 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520' 2024-09-10T21:10:34.578 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612628 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612628 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628' 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182467 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182467 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467' 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467 5-420906795010' 2024-09-10T21:10:34.579 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855414 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855414 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467 5-420906795010 6-150323855414' 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691883 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691883 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467 5-420906795010 6-150323855414 7-171798691883' 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:34.580 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:35.247 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clonehome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:35.247 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673046 2024-09-10T21:10:35.247 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673046 2024-09-10T21:10:35.247 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046' 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509522 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509522 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522' 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612629 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612629 2024-09-10T21:10:35.248 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629' 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182468 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182468 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468' 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468 5-420906795011' 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855415 2024-09-10T21:10:35.249 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855415 2024-09-10T21:10:35.250 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468 5-420906795011 6-150323855415' 2024-09-10T21:10:35.250 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.250 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_p.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528358 2024-09-10T21:10:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528358 2024-09-10T21:10:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467 5-420906795010 6-150323855414 7-171798691883 8-193273528358' 2024-09-10T21:10:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673044 2-64424509520 3-313532612628 4-107374182467 5-420906795010 6-150323855414 7-171798691883 8-193273528358 9-425201762306' 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836571 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836571 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836571 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.0 seq 21474836571 2024-09-10T21:10:35.434 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836571 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673044 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673044 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673044 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.1 seq 42949673044 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673044 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:35.435 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-10T21:10:35.436 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:35.436 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.cg_stats: seq=171798691884 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468 5-420906795011 6-150323855415 7-171798691884' 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528359 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528359 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468 5-420906795011 6-150323855415 7-171798691884 8-193273528359' 2024-09-10T21:10:35.802 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762307 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762307 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509522 3-313532612629 4-107374182468 5-420906795011 6-150323855415 7-171798691884 8-193273528359 9-425201762307' 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-10T21:10:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.0 seq 21474836572 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673046 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673046 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673046 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-10T21:10:35.804 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.1 seq 42949673046 2024-09-10T21:10:36.331 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stlient.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-10T21:10:36.331 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.2 seq 64424509520 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509520 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612628 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612628 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612628 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-09-10T21:10:36.332 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.3 seq 313532612628 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612628 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182467 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182467 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182467 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.4 seq 107374182467 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:36.333 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182467 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-420906795010 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-420906795010 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:36.334 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waitiats: ceph osd last-stat-seq 1 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509522 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509522 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509522 2024-09-10T21:10:37.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.2 seq 64424509522 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509522 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612629 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612629 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612629 2024-09-10T21:10:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612629' 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.3 seq 313532612629 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612629 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182468 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182468 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182468 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-09-10T21:10:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.4 seq 107374182468 2024-09-10T21:10:37.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:37.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182468 2024-09-10T21:10:37.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-420906795011 2024-09-10T21:10:37.570 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cepng osd.5 seq 420906795010' 2024-09-10T21:10:37.570 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.5 seq 420906795010 2024-09-10T21:10:37.570 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:10:37.570 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795010 2024-09-10T21:10:37.570 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855414 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855414 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855414 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.6 seq 150323855414 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855415 -lt 150323855414 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.571 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691883 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691883 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691883 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.7 seq 171798691883 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691884 -lt 171798691883 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528358 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:37.572 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528358 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528358 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.8 seq 193273528358 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:10:37.573 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528358 2024-09-10T21:10:38.015 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pghtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:38.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:10:38.015 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-420906795011 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 420906795011' 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.5 seq 420906795011 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795011 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855415 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855415 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:38.016 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855415 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.6 seq 150323855415 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855415 -lt 150323855415 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691884 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691884 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.7 seq 171798691884 2024-09-10T21:10:38.017 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691884 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528359 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:10:38.018 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528359 2024-09-10T21:10:38.978 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cu_stats: for s in $seqs 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-425201762306 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-425201762306 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 425201762306' 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: waiting osd.9 seq 425201762306 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762306 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:10:38.979 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:38.980 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:38.981 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:39.619 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubut -d - -f 2 2024-09-10T21:10:39.619 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528359 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.8 seq 193273528359 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528359 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-425201762307 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-425201762307 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762307 2024-09-10T21:10:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 425201762307' 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: waiting osd.9 seq 425201762307 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762307 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:39.621 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:247994: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:10:39.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:10:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:247992: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 247990 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:10:39.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:10:39.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:39.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:40.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:40.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:10:40.484 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:40.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:40.498 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:40.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:06.464927+0000 '>' 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:40.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:41.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:41.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:41.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:42.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:06.464927+0000 '>' 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:42.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:43.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:43.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:43.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:43.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:43.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:43.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:43.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:06.464927+0000 '>' 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:43.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:44.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:44.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:44.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:44.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:44.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:44.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:44.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:44.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:06.464927+0000 '>' 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:44.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:10:45.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:10:45.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:10:45.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:10:45.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:10:45.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:10:45.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:10:45.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:10:45.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:41.159236+0000 '>' 2024-09-10T21:10:06.464927+0000 2024-09-10T21:10:45.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:10:45.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:10:47.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T21:10:47.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:10:47.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 251474"' 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 251474' 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 251475"' 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 251475' 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 251474 251475' 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:10:47.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:10:47.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 251474 2024-09-10T21:10:47.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/251477: /' 2024-09-10T21:10:47.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:47.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/251479: /' 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.068 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:10:49.069 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: _ 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: hinfo_key 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: snapset 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:10:49.070 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:10:49.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:10:49.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:10:49.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: start osd.5 2024-09-10T21:10:49.108 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.108 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:10:49.108 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:10:49.109 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: _ 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: hinfo_key 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: snapset 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:10:49.110 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:10:49.111 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:49.112 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T21:10:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:251479: start osd.9 2024-09-10T21:10:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:10:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:10:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:10:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:10:53.898 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2024-09-10T21:10:49.098+0000 7f29d4c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2024-09-10T21:10:49.100+0000 7f29d4c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2024-09-10T21:10:49.101+0000 7f29d4c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 0 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2024-09-10T21:10:50.461+0000 7f29d4c18640 -1 Falling back to public interface 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 1 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2024-09-10T21:10:52.910+0000 7f29d4c18640 -1 osd.5 102 log_to_monitors true 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 3 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251477: osd.5 up in weight 1 up_from 106 up_thru 98 down_very_ops 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2024-09-10T21:10:49.138+0000 7f8fa740f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.902 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2024-09-10T21:10:49.289+0000 7f8fa740f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2024-09-10T21:10:49.291+0000 7f8fa740f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.903 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 0 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2024-09-10T21:10:50.419+0000 7f8fa740f640 -1 Falling back to public interface 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:10:53.904 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 1 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:53.905 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2024-09-10T21:10:53.400+0000 7f8fa740f640 -1 osd.9 102 log_to_monitors true 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 3 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:53.906 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:55.070 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cat 103 last_clean_interval [98,102) [v2:127.0.0.1:6874/4079203732,v1:127.0.0.1:6875/4079203732] [v2:127.0.0.1:6876/4079203732,v1:127.0.0.1:6877/4079203732] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:10:55.071 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 1 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 2 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 3 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 4 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 5 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 6 2024-09-10T21:10:55.072 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 7 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 8 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: 9' 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051' 2024-09-10T21:10:55.073 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.079 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/cloeph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:10:56.079 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:10:56.079 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:10:56.079 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 4 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: osd.9 up in weight 1 up_from 109 up_thru 0 down_at 103 last_clean_interval [99,102) [v2:127.0.0.1:6842/2144155501,v1:127.0.0.1:6843/2144155501] [v2:127.0.0.1:6844/2144155501,v1:127.0.0.1:6845/2144155501] exists,up 4022b805-5ff0-4b04-ac5e-8f8ae621adfd 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:10:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:251479: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 1 2024-09-10T21:10:56.081 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 2 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 3 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 4 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 5 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 6 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 7 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 8 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: 9' 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836578 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836578 2024-09-10T21:10:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578' 2024-09-10T21:10:56.507 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ne.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527' 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612634 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612634 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634' 2024-09-10T21:10:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182473 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182473 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473' 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473 5-455266533378' 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.509 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855420 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855420 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473 5-455266533378 6-150323855420' 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691890 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691890 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473 5-455266533378 6-150323855420 7-171798691890' 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:56.510 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:57.058 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/ids 2024-09-10T21:10:57.058 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:10:57.058 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673052 2024-09-10T21:10:57.058 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673052 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052' 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509528 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509528 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528' 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612635 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612635 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635' 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:10:57.059 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474' 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474 5-455266533379' 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855421 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855421 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474 5-455266533379 6-150323855421' 2024-09-10T21:10:57.060 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.061 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528365 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528365 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473 5-455266533378 6-150323855420 7-171798691890 8-193273528365' 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435266 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435266 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-313532612634 4-107374182473 5-455266533378 6-150323855420 7-171798691890 8-193273528365 9-468151435266' 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:57.236 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.0 seq 21474836577 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:57.237 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.1 seq 42949673051 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:57.238 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:57.616 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/ce:2210: flush_pg_stats: seq=171798691891 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691891 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474 5-455266533379 6-150323855421 7-171798691891' 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528366 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528366 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474 5-455266533379 6-150323855421 7-171798691891 8-193273528366' 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435267 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435267 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673052 2-64424509528 3-313532612635 4-107374182474 5-455266533379 6-150323855421 7-171798691891 8-193273528366 9-468151435267' 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836578 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836578 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836578 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.0 seq 21474836578 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836578 -lt 21474836578 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673052 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:10:57.618 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673052 2024-09-10T21:10:57.619 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:57.619 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673052 2024-09-10T21:10:57.619 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-09-10T21:10:57.619 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.1 seq 42949673052 2024-09-10T21:10:58.302 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-10T21:10:58.302 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:58.302 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.2 seq 64424509527 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509528 -lt 64424509527 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612634 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612634 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612634 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612634' 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.3 seq 313532612634 2024-09-10T21:10:58.303 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612635 -lt 313532612634 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182473 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182473 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182473 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.4 seq 107374182473 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182474 -lt 107374182473 2024-09-10T21:10:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533378 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533378 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:58.305 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats2: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673052 -lt 42949673052 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509528 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509528 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.116 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509528 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.2 seq 64424509528 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509528 -lt 64424509528 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612635 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612635 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612635 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.3 seq 313532612635 2024-09-10T21:10:59.117 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612635 -lt 313532612635 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.4 seq 107374182474 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182474 -lt 107374182474 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.118 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533379 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //h: echo 'waiting osd.5 seq 455266533378' 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.5 seq 455266533378 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533379 -lt 455266533378 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855420 2024-09-10T21:10:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855420 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855420 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.6 seq 150323855420 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855421 -lt 150323855420 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691890 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691890 2024-09-10T21:10:59.557 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691890 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.7 seq 171798691890 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528365 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528365 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528365 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.8 seq 193273528365 2024-09-10T21:10:59.558 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:10:59.559 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-10T21:11:00.018 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533379 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.5 seq 455266533379 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-10T21:11:00.019 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855421 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855421 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855421 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.6 seq 150323855421 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855422 -lt 150323855421 2024-09-10T21:11:00.020 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691891 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691891 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691891 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.7 seq 171798691891 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691891 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:00.021 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528366 2024-09-10T21:11:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:11:00.022 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528366 2024-09-10T21:11:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435266 2024-09-10T21:11:00.880 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435266 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435266 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435266' 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: waiting osd.9 seq 468151435266 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435267 -lt 468151435266 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:00.881 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:251477: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:00.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:11:00.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 251475 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:_pg_stats: cut -d - -f 2 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528366 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.8 seq 193273528366 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528366 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435267 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435267 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435267 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435267' 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: waiting osd.9 seq 468151435267 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435267 -lt 468151435267 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:251479: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:01.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:11:01.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:11:01.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:11:01.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T21:11:01.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:11:01.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:11:01.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:11:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-10T21:11:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-10T21:11:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-10T21:11:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-10T21:11:01.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254369"' 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254369' 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254371"' 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254371' 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-10T21:11:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 254369 254371' 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254369 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254372: /' 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:11:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254374: /' 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:11:04.887 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:11:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:11:04.889 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:11:04.890 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:11:04.890 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:04.891 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: start osd.3 2024-09-10T21:11:04.892 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:2254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:11:05.329 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-10T21:11:05.330 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:11:05.331 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:05.332 INFO:tasks.workunit.client.0.smithi181.stderr:254374: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:11:05.333 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: start osd.5 2024-09-10T21:11:05.334 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:11:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:11:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:11:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:11:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:11:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2024-09-10T21:11:04.918+0000 7f4e56883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2024-09-10T21:11:04.920+0000 7f4e56883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2024-09-10T21:11:04.922+0000 7f4e56883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 0 2024-09-10T21:11:09.456 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2024-09-10T21:11:06.062+0000 7f4e56883640 -1 Falling back to public interface 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 1 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.457 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2024-09-10T21:11:07.895+0000 7f4e56883640 -1 osd.3 110 log_to_monitors true 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 3 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.458 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:254372: osd.3 up in weight 1 up_from 115 up_thru 115 down_at 111 la54374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:11:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:11:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:11:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2024-09-10T21:11:05.354+0000 7f74224dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2024-09-10T21:11:05.356+0000 7f74224dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2024-09-10T21:11:05.357+0000 7f74224dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 0 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.986 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2024-09-10T21:11:06.707+0000 7f74224dc640 -1 Falling back to public interface 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 1 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2024-09-10T21:11:08.815+0000 7f74224dc640 -1 osd.5 111 log_to_monitors true 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 3 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254374: osd.5 up in weight 1 up_from 118 up_thru 106 down_at 111 last_clean_interval [73,110) [v2:127.0.0.1:6826/2662838229,v1:127.0.0.1:6827/2662838229] [v2:127.0.0.1:6828/2662838229,v1:127.0.0.1:6829/2662838229] exists,up 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:11:10.553 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:11:10.554 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 1 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 2 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 3 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 4 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 5 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 6 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 7 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 8 2024-09-10T21:11:10.555 INFO:tasks.workunit.client.0.smithi181.stderr:254372: 9' 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836582 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836582 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582' 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056' 2024-09-10T21:11:10.556 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.clientst_clean_interval [106,111) [v2:127.0.0.1:6874/2444628127,v1:127.0.0.1:6875/2444628127] [v2:127.0.0.1:6876/2444628127,v1:127.0.0.1:6877/2444628127] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:11:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:254374: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 1 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 2 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 3 2024-09-10T21:11:11.176 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 4 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 5 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 6 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 7 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 8 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: 9' 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836583 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836583 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583' 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:11:11.177 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-10T21:11:11.178 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-10T21:11:11.178 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057' 2024-09-10T21:11:11.178 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.206 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.clien.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532' 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042' 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:11:12.207 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182478 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182478 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478' 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140930 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140930 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478 5-506806140930' 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855425 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855425 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478 5-506806140930 6-150323855425' 2024-09-10T21:11:12.208 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691895 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691895 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478 5-506806140930 6-150323855425 7-171798691895' 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.209 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/t.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509533 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509533 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533' 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239043 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239043 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043' 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.424 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182479 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182479 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479' 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140931 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140931 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479 5-506806140931' 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855426 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855426 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479 5-506806140931 6-150323855426' 2024-09-10T21:11:12.425 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691896 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691896 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479 5-506806140931 6-150323855426 7-171798691896' 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.426 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtestclone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528370 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528370 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478 5-506806140930 6-150323855425 7-171798691895 8-193273528370' 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435271 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435271 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-493921239042 4-107374182478 5-506806140930 6-150323855425 7-171798691895 8-193273528370 9-468151435271' 2024-09-10T21:11:12.982 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836582 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836582 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836582 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.0 seq 21474836582 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836582 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:11:12.983 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.1 seq 42949673056 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673057 -lt 42949673056 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:12.984 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/cl/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528371 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528371 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479 5-506806140931 6-150323855426 7-171798691896 8-193273528371' 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435272 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435272 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673057 2-64424509533 3-493921239043 4-107374182479 5-506806140931 6-150323855426 7-171798691896 8-193273528371 9-468151435272' 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836583 2024-09-10T21:11:13.129 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836583 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836583 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.0 seq 21474836583 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836583 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-10T21:11:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.1 seq 42949673057 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673057 -lt 42949673057 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509533 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.131 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:11:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/cone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-10T21:11:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.2 seq 64424509532 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239042 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239042 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-10T21:11:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.3 seq 493921239042 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239044 -lt 493921239042 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182478 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182478 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182478 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.4 seq 107374182478 2024-09-10T21:11:13.880 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182479 -lt 107374182478 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-506806140930 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-506806140930 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:13.881 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140930 2024-09-10T21:11:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'lone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509533 2024-09-10T21:11:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509533 2024-09-10T21:11:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.2 seq 64424509533 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509533 -lt 64424509533 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239043 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239043 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239043 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.3 seq 493921239043 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:11:14.051 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239044 -lt 493921239043 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182479 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182479 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182479 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.4 seq 107374182479 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182479 -lt 107374182479 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-506806140931 2024-09-10T21:11:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:11:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-506806140931 2024-09-10T21:11:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:14.053 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140931 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo waiting osd.5 seq 506806140930' 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.5 seq 506806140930 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140930 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855425 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:11:15.089 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855425 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855425 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.6 seq 150323855425 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855427 -lt 150323855425 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691895 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691895 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691895 2024-09-10T21:11:15.090 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.7 seq 171798691895 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691895 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528370 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528370 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528370 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.8 seq 193273528370 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:11:15.091 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528370 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flu'waiting osd.5 seq 506806140931' 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.5 seq 506806140931 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140931 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855426 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.239 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855426 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855426 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.6 seq 150323855426 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855427 -lt 150323855426 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691896 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691896 2024-09-10T21:11:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691896 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.7 seq 171798691896 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691896 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528371 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528371 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528371 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-10T21:11:15.241 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.8 seq 193273528371 2024-09-10T21:11:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:11:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528371 2024-09-10T21:11:16.379 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flsh_pg_stats: for s in $seqs 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435271 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435271 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435271 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435271' 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: waiting osd.9 seq 468151435271 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435273 -lt 468151435271 2024-09-10T21:11:16.380 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:254372: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:254372: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:11:16.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254371 2024-09-10T21:11:16.531 INFO:tasks.workunit.client.0.smithi181.stderr:ush_pg_stats: for s in $seqs 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435272 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435272 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435272 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435272' 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: waiting osd.9 seq 468151435272 2024-09-10T21:11:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435273 -lt 468151435272 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:16.533 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:254374: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:11:16.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:11:16.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:11:16.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:11:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:17.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:11:17.392 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-10T21:11:17.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:11:17.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:11:17.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:41.159236+0000 '>' 2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:17.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:11:18.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:11:18.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:11:18.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:11:18.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:11:18.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:11:18.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:11:18.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:11:18.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:10:41.159236+0000 '>' 2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:18.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:11:19.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:11:19.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:11:19.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:11:19.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:11:19.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:11:19.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:11:19.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:11:20.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:11:17.878379+0000 '>' 2024-09-10T21:10:41.159236+0000 2024-09-10T21:11:20.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:11:20.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:11:22.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-10T21:11:22.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:11:22.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 257730"' 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 257730' 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 257731"' 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 257731' 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 257730 257731' 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 257730 2024-09-10T21:11:22.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:11:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/257733: /' 2024-09-10T21:11:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:11:22.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/257735: /' 2024-09-10T21:11:23.336 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.336 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:23.337 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: _ 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: hinfo_key 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: snapset 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:11:23.338 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' 257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.339 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:23.340 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: _ 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: hinfo_key 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: snapset 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:11:23.341 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper--osd-failsafe-full-ratio=.99' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:11:23.342 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:11:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: start osd.3 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-ls.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:11:23.344 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:23.345 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T21:11:23.346 INFO:tasks.workunit.client.0.smithi181.stderr:257735: start osd.5 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:11:27.784 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2024-09-10T21:11:23.367+0000 7f78d7e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2024-09-10T21:11:23.370+0000 7f78d7e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2024-09-10T21:11:23.371+0000 7f78d7e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 0 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:27.785 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2024-09-10T21:11:24.742+0000 7f78d7e22640 -1 Falling back to public interface 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 1 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:27.786 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2024-09-10T21:11:27.198+0000 7f78d7e22640 -1 osd.3 120 log_to_monitors true 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 3 2024-09-10T21:11:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpervery_ops 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:11:28.244 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2024-09-10T21:11:23.371+0000 7f19f9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2024-09-10T21:11:23.374+0000 7f19f9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2024-09-10T21:11:23.375+0000 7f19f9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 0 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:28.245 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2024-09-10T21:11:24.976+0000 7f19f9013640 -1 Falling back to public interface 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 1 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:28.246 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2024-09-10T21:11:27.667+0000 7f19f9013640 -1 osd.5 120 log_to_monitors true 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 3 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:28.247 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257733: osd.3 up in weight 1 up_from 125 up_thru 118 down_at 121 last_clean_interval [115,120) [v2:127.0.0.1:6826/1726178375,v1:127.0.0.1:6827/1726178375] [v2:127.0.0.1:6828/1726178375,v1:127.0.0.1:6829/1726178375] exists,up 1852b6c3-b374-40ec-b76e-f9ab64ef4e75 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:11:29.193 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:11:29.194 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 1 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 2 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 3 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 4 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 5 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 6 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 7 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 8 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: 9' 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:11:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836588 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836588 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588' 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-10T21:11:29.196 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061' 2024-09-10T21:11:30.777 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/eph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 4 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: osd.5 up in weight 1 up_from 128 up_thru 118 down_at 121 last_clean_interval [118,120) [v2:127.0.0.1:6874/3175548358,v1:127.0.0.1:6875/3175548358] [v2:127.0.0.1:6876/3175548358,v1:127.0.0.1:6877/3175548358] exists,up aed94797-5266-4012-90c9-80165a1793b3 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:11:30.778 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:11:30.779 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 1 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 2 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 3 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 4 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 5 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 6 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 7 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 8 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 9' 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.780 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836589 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836589 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589' 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd iqa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509538 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509538 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538' 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=536870912002 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 536870912002 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002' 2024-09-10T21:11:30.781 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182484 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182484 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484' 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813890 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813890 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484 5-549755813890' 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855431 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855431 2024-09-10T21:11:30.782 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484 5-549755813890 6-150323855431' 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691900 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691900 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484 5-549755813890 6-150323855431 7-171798691900' 2024-09-10T21:11:30.783 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qan $ids 2024-09-10T21:11:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:11:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673063 2024-09-10T21:11:31.957 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673063 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063' 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509539 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509539 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539' 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=536870912003 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 536870912003 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003' 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T21:11:31.958 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182485 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182485 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485' 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813891 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813891 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485 5-549755813891' 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855432 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855432 2024-09-10T21:11:31.959 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485 5-549755813891 6-150323855432' 2024-09-10T21:11:31.960 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:31.960 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528376 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528376 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484 5-549755813890 6-150323855431 7-171798691900 8-193273528376' 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435277 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435277 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836588 1-42949673061 2-64424509538 3-536870912002 4-107374182484 5-549755813890 6-150323855431 7-171798691900 8-193273528376 9-468151435277' 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836588 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836588 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836588 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.0 seq 21474836588 2024-09-10T21:11:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836589 -lt 21474836588 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.1 seq 42949673061 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673061 2024-09-10T21:11:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:32.130 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509538 2024-09-10T21:11:32.130 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/ceph.sh:2210: flush_pg_stats: seq=171798691902 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691902 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485 5-549755813891 6-150323855432 7-171798691902' 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528377 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528377 2024-09-10T21:11:32.560 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485 5-549755813891 6-150323855432 7-171798691902 8-193273528377' 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=468151435278 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 468151435278 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509539 3-536870912003 4-107374182485 5-549755813891 6-150323855432 7-171798691902 8-193273528377 9-468151435278' 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836589 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836589 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836589 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-10T21:11:32.561 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.0 seq 21474836589 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673063 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673063 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673063 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-10T21:11:32.562 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.1 seq 42949673063 2024-09-10T21:11:33.066 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:test/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509538 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509538 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.2 seq 64424509538 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509538 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-536870912002 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:11:33.067 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-536870912002 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=536870912002 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 536870912002' 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.3 seq 536870912002 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 536870912002 -lt 536870912002 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182484 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182484 2024-09-10T21:11:33.068 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182484 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.4 seq 107374182484 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182484 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-549755813890 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-549755813890 2024-09-10T21:11:33.069 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.897 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549752222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:11:33.897 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673063 2024-09-10T21:11:33.897 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.897 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509539 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509539 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509539 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.2 seq 64424509539 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509539 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.898 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-536870912003 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-536870912003 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=536870912003 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 536870912003' 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.3 seq 536870912003 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 536870912004 -lt 536870912003 2024-09-10T21:11:33.899 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182485 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182485 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182485 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.4 seq 107374182485 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182485 -lt 107374182485 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:33.900 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-549755813891 2024-09-10T21:11:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:257735: 5813890 2024-09-10T21:11:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 549755813890' 2024-09-10T21:11:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.5 seq 549755813890 2024-09-10T21:11:34.330 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813891 -lt 549755813890 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855431 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855431 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855431 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.6 seq 150323855431 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:11:34.331 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855433 -lt 150323855431 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691900 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691900 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691900 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.7 seq 171798691900 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691902 -lt 171798691900 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528376 2024-09-10T21:11:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528376 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528376 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.8 seq 193273528376 2024-09-10T21:11:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:11:34.817 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 19//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-549755813891 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813891 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 549755813891' 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.5 seq 549755813891 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813891 -lt 549755813891 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855432 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T21:11:34.818 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855432 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855432 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.6 seq 150323855432 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855433 -lt 150323855432 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691902 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691902 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:34.819 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691902 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.7 seq 171798691902 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691902 -lt 171798691902 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528377 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T21:11:34.820 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528377 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: fl3273528377 -lt 193273528376 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435277 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435277 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:35.648 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435277 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435277' 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: waiting osd.9 seq 468151435277 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435278 -lt 468151435277 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:35.649 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:257733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:11:35.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 257731 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:ush_pg_stats: cut -d - -f 2 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528377 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.8 seq 193273528377 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T21:11:36.440 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528377 -lt 193273528377 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-468151435278 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-468151435278 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=468151435278 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 468151435278' 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: waiting osd.9 seq 468151435278 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 468151435279 -lt 468151435278 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:11:36.441 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:11:36.442 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:257735: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-10T21:11:36.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:11:36.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:36.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:36.626 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:11:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:11:36.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:11:36.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:11:36.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:11:36.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:11:36.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:11:36.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:11:36.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:11:36.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:11:36.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:11:36.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:11:36.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:11:36.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:11:36.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:36.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:36.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:11:36.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:11:36.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:11:36.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:11:36.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:11:36.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:11:36.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:11:36.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:11:36.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:11:36.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:11:36.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:11:36.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:11:36.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:11:36.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:11:36.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:11:36.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:11:36.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:11:36.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:11:36.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:11:36.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:11:36.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:11:36.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:11:36.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:11:36.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:36.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:36.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:11:36.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:11:36.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:11:36.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:11:36.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:11:36.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:36.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:36.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:11:36.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:11:36.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-10T21:11:37.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:11:37.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:37.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:37.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:37.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:37.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:37.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:37.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:11:37.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:11:37.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:11:37.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:11:37.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:11:37.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:11:37.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:37.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:11:37.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:11:37.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:11:37.445 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:11:37.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:11:37.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:11:38.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:38.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:11:38.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:11:38.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:11:38.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:11:38.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:11:38.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:11:38.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab2bd43b-0e03-48da-8b66-3686f2885ca2 2024-09-10T21:11:38.377 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 ab2bd43b-0e03-48da-8b66-3686f2885ca2 2024-09-10T21:11:38.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ab2bd43b-0e03-48da-8b66-3686f2885ca2' 2024-09-10T21:11:38.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:11:38.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKtuBmNM9DFxAAL9UuKvNuA7gqJItftY9fOg== 2024-09-10T21:11:38.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKtuBmNM9DFxAAL9UuKvNuA7gqJItftY9fOg=="}' 2024-09-10T21:11:38.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab2bd43b-0e03-48da-8b66-3686f2885ca2 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:11:38.604 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:11:38.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:11:38.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKtuBmNM9DFxAAL9UuKvNuA7gqJItftY9fOg== --osd-uuid ab2bd43b-0e03-48da-8b66-3686f2885ca2 2024-09-10T21:11:38.644 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:38.640+0000 7ff6acbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:38.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:38.651+0000 7ff6acbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:38.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:38.652+0000 7ff6acbc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:38.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:38.652+0000 7ff6acbc5640 -1 bdev(0x55fbbb139c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:11:38.656 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:38.652+0000 7ff6acbc5640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:11:44.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:11:44.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:11:44.360 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:11:44.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:11:44.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:11:44.735 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:11:44.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:11:44.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:11:44.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:11:44.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:11:44.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:11:44.762 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:44.758+0000 7ff457e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:44.764 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:44.760+0000 7ff457e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:44.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:44.761+0000 7ff457e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:11:45.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:11:45.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:11:45.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:11:45.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:45.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:45.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:46.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:46.135+0000 7ff457e12640 -1 Falling back to public interface 2024-09-10T21:11:46.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:46.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:46.334 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:11:46.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:11:46.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:46.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:46.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:47.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:11:47.607+0000 7ff457e12640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:11:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:47.618 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:11:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:11:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:48.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:49.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:49.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:49.041 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:11:49.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:11:49.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:49.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:49.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:50.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:50.347 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:11:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:11:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:50.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:51.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:51.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:51.652 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:11:51.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:11:51.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:51.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:51.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:52.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:52.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:52.940 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:11:52.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:11:52.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:52.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:53.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:54.226 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:11:54.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:11:54.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:54.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:54.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:55.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:55.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:55.530 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:11:55.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:11:55.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:55.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:55.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:56.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:56.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:56.818 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:11:56.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:11:56.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:56.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:57.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:58.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:58.166 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:11:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:11:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:58.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:11:59.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:11:59.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:11:59.484 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:11:59.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:11:59.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:11:59.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:11:59.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:00.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:00.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:00.982 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:12:00.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:12:00.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:00.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:01.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:02.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:02.419 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:12:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:12:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:02.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:03.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:04.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:04.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:04.112 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:12:04.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:12:04.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:04.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:04.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:05.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:05.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:05.566 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:12:05.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:12:05.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:05.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:06.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:07.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:07.015 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:12:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:12:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:07.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:07.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:08.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:08.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:08.652 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:12:08.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:12:08.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:08.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:09.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:10.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:10.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:10.606 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:12:10.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:12:10.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:10.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:10.928 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1549658752,v1:127.0.0.1:6803/1549658752] [v2:127.0.0.1:6804/1549658752,v1:127.0.0.1:6805/1549658752] exists,up ab2bd43b-0e03-48da-8b66-3686f2885ca2 2024-09-10T21:12:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:12:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:12:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:12:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-09-10T21:12:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:12:10.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:12:10.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:12:10.935 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:12:10.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c242875f-5044-4860-9cd9-6cf4d5c9a381' 2024-09-10T21:12:10.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:12:10.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqtuBmUmd2OBAAL54ZiUiLc5JsaKaunz5cCw== 2024-09-10T21:12:10.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqtuBmUmd2OBAAL54ZiUiLc5JsaKaunz5cCw=="}' 2024-09-10T21:12:10.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c242875f-5044-4860-9cd9-6cf4d5c9a381 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:12:11.317 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:12:11.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:12:11.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqtuBmUmd2OBAAL54ZiUiLc5JsaKaunz5cCw== --osd-uuid c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:12:11.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:11.355+0000 7fab1ea18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:11.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:11.397+0000 7fab1ea18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:11.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:11.398+0000 7fab1ea18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:11.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:11.399+0000 7fab1ea18640 -1 bdev(0x5612df89fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:12:11.403 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:11.399+0000 7fab1ea18640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:12:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:12:17.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:12:17.424 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:12:17.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:12:17.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:12:17.783 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:12:17.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:12:17.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:12:17.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:12:17.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:12:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:12:17.811 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:17.807+0000 7f7fe300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:17.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:17.809+0000 7f7fe300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:17.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:17.810+0000 7f7fe300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:18.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:12:18.101 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:12:18.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:18.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:18.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:19.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:19.148+0000 7f7fe300e640 -1 Falling back to public interface 2024-09-10T21:12:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:19.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:12:19.428 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:12:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:19.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:19.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:20.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:20.710 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:12:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:12:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:20.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:20.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:21.123 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:21.119+0000 7f7fe300e640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:12:21.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:21.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:21.996 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:12:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:12:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:22.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:23.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:23.387 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:12:23.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:12:23.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:23.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:23.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:24.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:24.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:24.679 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:12:24.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:12:24.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:24.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:25.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:26.208 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:12:26.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:12:26.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:26.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:26.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:27.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:27.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:27.497 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:12:27.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:12:27.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:27.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:27.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:28.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:28.792 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:12:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:12:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:28.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:29.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:30.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:30.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:30.086 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:12:30.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:12:30.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:30.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:30.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:31.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:31.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:31.375 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:12:31.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:12:31.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:31.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:31.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:32.662 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:12:32.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:12:32.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:32.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:32.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:33.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:33.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:33.956 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:12:33.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:12:33.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:33.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:34.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:35.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:35.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:35.577 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:12:35.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:12:35.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:35.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:36.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:37.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:37.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:37.226 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:12:37.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:12:37.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:37.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:37.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:38.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:38.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:38.848 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:12:38.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:12:38.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:38.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:39.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:40.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:40.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:40.323 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:12:40.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:12:40.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:40.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:40.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:41.779 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:12:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:12:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:41.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:42.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:43.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:43.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:43.371 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:12:43.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:12:43.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:43.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:43.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:44.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:44.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:44.664 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:12:44.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:12:44.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:44.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:12:44.974 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1791936253,v1:127.0.0.1:6811/1791936253] [v2:127.0.0.1:6812/1791936253,v1:127.0.0.1:6813/1791936253] exists,up c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:12:44.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:12:44.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:12:44.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:12:44.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-09-10T21:12:44.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:12:45.244 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:12:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:12:45.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:12:45.633 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:12:45.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:12:46.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:12:48.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:12:48.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:12:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:12:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:12:48.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:12:48.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:12:48.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:12:49.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-10T21:12:49.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-10T21:12:49.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-10T21:12:49.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:12:49.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496 1-42949672966' 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-10T21:12:49.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:12:49.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:12:49.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-10T21:12:49.203 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:12:49.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-10T21:12:49.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-10T21:12:49.204 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836496 2024-09-10T21:12:49.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:12:49.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-10T21:12:49.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:12:50.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:12:50.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:12:50.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836496 2024-09-10T21:12:50.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:12:50.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-10T21:12:50.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:12:50.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:12:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-10T21:12:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:12:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-10T21:12:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-10T21:12:50.772 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672966 2024-09-10T21:12:50.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:12:51.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672966 2024-09-10T21:12:51.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:12:51.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:12:51.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:12:51.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:12:51.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:12:51.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:12:51.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:12:51.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:12:51.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:12:51.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:12:52.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:12:52.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:12:52.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:12:52.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:12:52.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:12:52.419 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:12:52.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:12:52.980 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:12:52.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:12:52.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:12:52.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:12:53.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-09-10T21:12:53.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-10T21:12:53.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T21:12:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-10T21:12:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T21:12:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:12:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T21:12:53.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:12:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:12:53.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T21:12:53.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:12:53.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:12:53.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:12:53.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:12:54.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:12:54.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T21:12:55.751 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T21:12:56.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:12:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:12:56.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:12:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:12:56.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:12:56.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:56.297+0000 7f8c1a40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:56.304 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:56.299+0000 7f8c1a40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:56.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:56.300+0000 7f8c1a40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:12:56.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:56.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:56.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:57.179 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:57.175+0000 7f8c1a40b640 -1 Falling back to public interface 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:58.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:12:58.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:12:58.991+0000 7f8c1a40b640 -1 osd.0 21 log_to_monitors true 2024-09-10T21:12:59.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:12:59.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:12:59.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:12:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:12:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:12:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:12:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 26 up_thru 12 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1093622688,v1:127.0.0.1:6803/1093622688] [v2:127.0.0.1:6804/1093622688,v1:127.0.0.1:6805/1093622688] exists,up ab2bd43b-0e03-48da-8b66-3686f2885ca2 2024-09-10T21:12:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:12:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:12:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:12:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:12:59.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:12:59.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:13:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:13:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:13:00.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:13:00.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:00.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:13:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-10T21:13:00.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-10T21:13:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-10T21:13:00.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:00.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:13:00.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-10T21:13:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-10T21:13:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-09-10T21:13:00.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:00.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-10T21:13:00.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:00.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:13:00.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-10T21:13:00.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:00.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-10T21:13:00.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-10T21:13:00.479 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149698 2024-09-10T21:13:00.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:00.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-10T21:13:00.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:00.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-10T21:13:00.756 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:00.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:13:00.757 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-10T21:13:00.757 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:00.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-10T21:13:00.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-10T21:13:00.758 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672970 2024-09-10T21:13:00.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:01.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672970 2024-09-10T21:13:01.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:13:02.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:13:02.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:02.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672970 2024-09-10T21:13:02.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:13:03.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:13:03.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:03.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-10T21:13:03.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:13:03.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:03.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:03.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:13:04.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:04.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:04.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:04.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-10T21:13:04.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:13:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-10T21:13:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-10T21:13:04.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-10T21:13:04.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-10T21:13:04.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:04.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:04.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:04.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:05.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:05.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-10T21:13:05.420 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:05.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:05.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:05.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:05.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:12:45.577986+0000 '>' 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:05.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:06.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:06.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:06.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:06.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:06.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:06.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:06.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:07.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:12:45.577986+0000 '>' 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:07.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:08.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:08.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:08.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:08.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:08.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:08.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:08.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:08.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:12:45.577986+0000 '>' 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:08.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:09.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:09.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:09.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:12:45.577986+0000 '>' 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:09.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:10.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:10.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:10.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:10.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:10.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:10.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:10.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:10.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:12:45.577986+0000 2024-09-10T21:13:10.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:13:10.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:13:12.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:12.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:13:12.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:13:12.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:13:12.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T21:13:13.562 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T21:13:13.562 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:13:13.562 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:13:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:13:13.832 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:13:13.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:13:13.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:13:13.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:13:13.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:13:13.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:13:13.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:13:13.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:13.857+0000 7f3b22811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:13.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:13.860+0000 7f3b22811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:13.861+0000 7f3b22811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:13:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:13:14.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:14.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:14.982 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:14.978+0000 7f3b22811640 -1 Falling back to public interface 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:15.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:13:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:13:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:16.715+0000 7f3b22811640 -1 osd.1 28 log_to_monitors true 2024-09-10T21:13:17.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:18.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:13:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:13:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:18.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:18.357 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/159943898,v1:127.0.0.1:6811/159943898] [v2:127.0.0.1:6812/159943898,v1:127.0.0.1:6813/159943898] exists,up c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:13:18.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:13:18.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:13:18.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:13:18.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:13:18.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:13:18.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:13:18.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:13:18.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:13:18.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:13:18.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:13:18.739 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:13:18.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:13:18.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:18.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:13:18.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-10T21:13:18.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-10T21:13:18.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-10T21:13:18.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:18.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:13:18.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-10T21:13:18.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-10T21:13:18.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-10T21:13:18.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:18.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-10T21:13:18.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:18.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:13:18.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-10T21:13:18.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:18.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-10T21:13:18.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-10T21:13:18.949 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149702 2024-09-10T21:13:18.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-10T21:13:19.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:13:20.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:13:20.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:20.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149703 -lt 111669149702 2024-09-10T21:13:20.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:20.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-10T21:13:20.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:20.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:13:20.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-10T21:13:20.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:20.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-10T21:13:20.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-10T21:13:20.531 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 137438953474 2024-09-10T21:13:20.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:20.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-10T21:13:20.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:13:20.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:20.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:21.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:21.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:21.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:13:21.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:21.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:21.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:13:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:13:21.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:13:21.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:13:21.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:13:21.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-09-10T21:13:21.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T21:13:21.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:13:21.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T21:13:21.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:13:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:13:22.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:13:22.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:13:22.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:13:22.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-10T21:13:22.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:13:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:13:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:13:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:13:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:22.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-10T21:13:23.709 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-10T21:13:24.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:13:24.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:13:24.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:13:24.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:13:24.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:13:24.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:13:24.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:13:24.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:13:24.233 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:13:24.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:13:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:24.257+0000 7f5beabfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:24.260+0000 7f5beabfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:24.261+0000 7f5beabfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:24.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:13:24.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:24.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:24.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:25.611+0000 7f5beabfd640 -1 Falling back to public interface 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:26.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:27.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:13:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:13:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:27.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:27.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:27.173+0000 7f5beabfd640 -1 osd.1 34 log_to_monitors true 2024-09-10T21:13:27.511 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 38 up_thru 32 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/1571667308,v1:127.0.0.1:6811/1571667308] [v2:127.0.0.1:6812/1571667308,v1:127.0.0.1:6813/1571667308] exists,up c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:13:27.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:13:27.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:13:27.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:13:27.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:13:27.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:13:27.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:13:27.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:13:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:13:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:13:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:13:27.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:27.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:13:28.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149705 2024-09-10T21:13:28.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149705 2024-09-10T21:13:28.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149705' 2024-09-10T21:13:28.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:28.069 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:13:28.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-10T21:13:28.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-10T21:13:28.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149705 1-163208757250' 2024-09-10T21:13:28.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:28.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149705 2024-09-10T21:13:28.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:28.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:13:28.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149705 2024-09-10T21:13:28.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:28.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149705 2024-09-10T21:13:28.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2024-09-10T21:13:28.226 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149705 2024-09-10T21:13:28.226 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:28.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149705 -lt 111669149705 2024-09-10T21:13:28.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:28.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757250 2024-09-10T21:13:28.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:28.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:13:28.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757250 2024-09-10T21:13:28.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:28.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-10T21:13:28.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-10T21:13:28.505 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 163208757250 2024-09-10T21:13:28.506 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:28.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-10T21:13:28.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:13:28.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:28.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:29.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:29.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T21:13:29.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:29.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:29.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:29.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 4 2024-09-10T21:13:29.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T21:13:29.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:13:29.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T21:13:29.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:13:29.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:13:29.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:13:29.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:29.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:29.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:29.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:29.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:29.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:29.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:30.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T21:13:30.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:30.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:30.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:30.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 4 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:13:30.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:13:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:13:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:13:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:13:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:13:30.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:13:31.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:13:31.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:31.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:31.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:13:31.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:31.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:31.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:31.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:13:31.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-10T21:13:31.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:31.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:32.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-10T21:13:32.476 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:13:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:32.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:32.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:32.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:32.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:32.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:32.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:32.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:33.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:33.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:33.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:33.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:33.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:34.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:34.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:35.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:35.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:35.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:35.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:35.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:35.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:35.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:35.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:35.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:36.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:36.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:36.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:36.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:36.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:36.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:36.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:37.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:37.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:37.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:37.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:37.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:37.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:37.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:37.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:06.109685+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:37.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:13:38.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:13:38.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:13:38.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:13:38.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:13:38.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:13:38.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:13:38.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:13:39.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:13:33.106815+0000 '>' 2024-09-10T21:13:06.109685+0000 2024-09-10T21:13:39.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:13:39.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:13:41.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:41.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:13:41.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:13:41.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:13:41.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:13:41.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:13:41.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-10T21:13:41.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:13:41.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:13:41.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:13:41.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:13:41.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:41.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-10T21:13:41.899 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-10T21:13:41.899 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:13:41.899 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:13:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:42.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:13:42.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:13:42.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:13:42.170 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:13:42.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:13:42.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:13:42.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:13:42.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:13:42.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:13:42.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:13:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:42.194+0000 7f8784f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:42.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:42.196+0000 7f8784f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:42.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:42.198+0000 7f8784f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:42.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:42.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:43.313 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:43.308+0000 7f8784f45640 -1 Falling back to public interface 2024-09-10T21:13:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:13:43.734 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:13:43.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:43.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:44.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:45.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:45.198 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:45.193+0000 7f8784f45640 -1 osd.1 39 log_to_monitors true 2024-09-10T21:13:45.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:13:46.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:13:46.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:46.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:13:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:13:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:46.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:13:46.726 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/467777580,v1:127.0.0.1:6811/467777580] [v2:127.0.0.1:6812/467777580,v1:127.0.0.1:6813/467777580] exists,up c242875f-5044-4860-9cd9-6cf4d5c9a381 2024-09-10T21:13:46.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:13:46.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:13:46.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:13:46.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:13:46.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:13:46.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:13:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:13:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:13:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:13:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:47.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:13:47.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149710 2024-09-10T21:13:47.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149710 2024-09-10T21:13:47.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149710' 2024-09-10T21:13:47.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:13:47.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:13:47.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-10T21:13:47.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-10T21:13:47.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149710 1-184683593730' 2024-09-10T21:13:47.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:47.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149710 2024-09-10T21:13:47.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:47.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:13:47.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149710 2024-09-10T21:13:47.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:47.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149710 2024-09-10T21:13:47.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149710' 2024-09-10T21:13:47.302 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149710 2024-09-10T21:13:47.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:47.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149709 -lt 111669149710 2024-09-10T21:13:47.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:13:48.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:13:48.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:13:48.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149710 -lt 111669149710 2024-09-10T21:13:48.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:13:48.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593730 2024-09-10T21:13:48.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:13:48.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:13:48.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593730 2024-09-10T21:13:48.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:13:48.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-10T21:13:48.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-09-10T21:13:48.867 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 184683593730 2024-09-10T21:13:48.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:13:49.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-10T21:13:49.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:13:49.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:49.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:49.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:13:49.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:13:49.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:13:49.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:13:49.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:13:49.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:13:49.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:13:49.525 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:13:49.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:13:49.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:13:49.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:13:49.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:13:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:13:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:13:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:13:50.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:13:50.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:13:50.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:13:50.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:13:50.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:13:50.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:13:50.279 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:13:50.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:13:50.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:13:50.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:13:50.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:13:50.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:13:50.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:13:50.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:13:50.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:13:50.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:13:50.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:13:50.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:13:50.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:50.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:50.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:13:50.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:13:50.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:13:50.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:13:50.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:13:50.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:13:50.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:13:50.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:13:50.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:13:50.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:13:50.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:13:50.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:13:50.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:13:50.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:13:50.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:13:50.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:13:50.308 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:13:50.308 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:50.308 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:50.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:13:50.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:13:50.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:13:50.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:13:50.310 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:13:50.310 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:50.310 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:50.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:13:50.312 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:13:50.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:13:50.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:13:50.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:13:50.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:13:50.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:13:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:13:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:50.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:50.771 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:50.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:50.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:13:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:13:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:13:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:13:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:13:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:13:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:13:50.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:13:50.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:13:50.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:13:50.974 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:13:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:13:50.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:13:50.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:13:51.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:51.058 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:51.059 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:13:51.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:13:51.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:13:51.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-10T21:13:51.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:13:51.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:13:51.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:13:51.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:13:51.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:13:51.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:13:51.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:51.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:13:51.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:13:51.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-10T21:13:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:13:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-10T21:13:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:13:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:13:51.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:13:51.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:13:51.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:13:51.978 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:13:51.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b0a4720a-7033-4624-854d-8b1ac2c4478b' 2024-09-10T21:13:51.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:13:51.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCPtuBm0sUJOxAAdil4ybPo9wxqXssJd0bQyQ== 2024-09-10T21:13:51.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCPtuBm0sUJOxAAdil4ybPo9wxqXssJd0bQyQ=="}' 2024-09-10T21:13:51.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b0a4720a-7033-4624-854d-8b1ac2c4478b -i td/osd-scrub-repair/0/new.json 2024-09-10T21:13:52.221 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:13:52.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:13:52.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPtuBm0sUJOxAAdil4ybPo9wxqXssJd0bQyQ== --osd-uuid b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:13:52.262 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:52.258+0000 7f6f6f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:52.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:52.268+0000 7f6f6f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:52.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:52.269+0000 7f6f6f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:52.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:52.270+0000 7f6f6f215640 -1 bdev(0x5615697d3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:13:52.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:52.270+0000 7f6f6f215640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:13:58.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:13:58.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:13:58.524 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:13:58.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:13:58.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:13:58.883 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:13:58.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:13:58.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:13:58.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:13:58.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:13:58.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:13:58.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:58.907+0000 7f4ac4e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:58.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:58.909+0000 7f4ac4e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:58.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:13:58.910+0000 7f4ac4e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:13:59.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:13:59.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:13:59.162 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:13:59.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:13:59.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:13:59.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:00.033 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:00.029+0000 7f4ac4e0d640 -1 Falling back to public interface 2024-09-10T21:14:00.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:00.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:00.439 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:14:00.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:14:00.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:00.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:00.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:01.599+0000 7f4ac4e0d640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:14:01.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:01.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:01.724 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:14:01.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:14:01.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:01.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:02.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:03.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:03.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:03.063 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:14:03.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:14:03.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:03.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:03.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:04.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:04.386 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:14:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:14:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:04.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:04.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:05.674 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:14:05.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:14:05.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:05.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:05.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:14:06.962 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:14:06.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:06.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:07.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:08.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:08.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:08.335 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:14:08.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:14:08.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:08.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:08.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:09.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:09.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:09.627 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:14:09.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:14:09.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:09.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:09.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:10.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:10.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:10.917 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:14:10.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:14:10.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:10.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:11.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:12.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:12.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:12.335 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:14:12.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:14:12.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:12.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:12.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:13.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:13.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:13.627 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:14:13.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:14:13.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:13.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:14.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:15.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:15.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:15.257 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:14:15.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:14:15.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:15.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:15.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:16.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:16.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:16.717 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:14:16.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:14:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:17.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:18.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:18.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:18.395 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:14:18.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:14:18.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:18.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:18.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:19.980 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:14:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:19.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:14:19.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:19.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:20.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:21.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:21.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:21.922 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:14:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:14:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:22.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:23.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:23.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:23.327 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:14:23.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:14:23.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:23.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:14:23.678 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1016190335,v1:127.0.0.1:6803/1016190335] [v2:127.0.0.1:6804/1016190335,v1:127.0.0.1:6805/1016190335] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:14:23.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:14:23.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:14:23.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:14:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1fad0485-4110-4f5e-8314-153221e152ff' 2024-09-10T21:14:23.686 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:14:23.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:14:23.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCvtuBmYTiFKRAARxdp70+CzKVAtnCN/ebLKA== 2024-09-10T21:14:23.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCvtuBmYTiFKRAARxdp70+CzKVAtnCN/ebLKA=="}' 2024-09-10T21:14:23.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1fad0485-4110-4f5e-8314-153221e152ff -i td/osd-scrub-repair/1/new.json 2024-09-10T21:14:24.128 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:14:24.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:14:24.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvtuBmYTiFKRAARxdp70+CzKVAtnCN/ebLKA== --osd-uuid 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:14:24.171 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:24.166+0000 7f7fc1862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:24.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:24.209+0000 7f7fc1862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:24.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:24.210+0000 7f7fc1862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:24.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:24.210+0000 7f7fc1862640 -1 bdev(0x561cbf669c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:14:24.214 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:24.210+0000 7f7fc1862640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:14:30.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:14:30.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:14:30.105 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:14:30.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:14:30.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:14:30.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:14:30.465 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:14:30.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:14:30.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:14:30.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:14:30.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:14:30.492 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:30.487+0000 7f8de5fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:30.495 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:30.490+0000 7f8de5fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:30.496 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:30.491+0000 7f8de5fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:14:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:30.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:14:30.745 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:14:30.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:30.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:31.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:31.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:31.852+0000 7f8de5fb2640 -1 Falling back to public interface 2024-09-10T21:14:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:14:32.022 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:14:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:32.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:32.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:33.366 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:14:33.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:14:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:33.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:33.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:33.685 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:33.680+0000 7f8de5fb2640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:14:34.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:34.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:34.649 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:14:34.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:14:34.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:34.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:34.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:35.983 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:14:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:14:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:35.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:36.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:37.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:37.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:37.361 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:14:37.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:14:37.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:37.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:37.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:38.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:38.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:38.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:14:38.772 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:14:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:38.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:39.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:40.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:40.097 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:14:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:14:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:40.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:40.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:41.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:41.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:41.387 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:14:41.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:14:41.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:41.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:41.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:42.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:42.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:42.784 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:14:42.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:14:42.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:42.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:43.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:44.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:44.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:44.074 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:14:44.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:14:44.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:44.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:44.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:45.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:45.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:45.411 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:14:45.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:14:45.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:45.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:45.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:46.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:46.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:46.815 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:14:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:14:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:46.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:47.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:48.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:48.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:48.128 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:14:48.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:14:48.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:48.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:48.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:49.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:49.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:49.930 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:14:49.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:14:49.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:49.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:50.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:51.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:51.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:51.394 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:14:51.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:14:51.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:51.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:51.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:52.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:52.904 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:14:52.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:14:52.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:52.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:53.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:54.206 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:14:54.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:14:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:54.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:54.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:55.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:55.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:55.692 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:14:55.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:14:55.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:55.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:56.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:14:57.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:14:57.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:14:57.094 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:14:57.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:14:57.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:14:57.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:14:57.420 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3529214936,v1:127.0.0.1:6811/3529214936] [v2:127.0.0.1:6812/3529214936,v1:127.0.0.1:6813/3529214936] exists,up 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:14:57.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:14:57.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:14:57.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:14:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:14:57.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:14:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:14:57.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5199b9c9-b511-43de-bc53-6987d1c26826' 2024-09-10T21:14:57.427 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:14:57.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:14:57.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRtuBmF84pGhAAGXPSuD45CDqTlqBtRtg5/w== 2024-09-10T21:14:57.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRtuBmF84pGhAAGXPSuD45CDqTlqBtRtg5/w=="}' 2024-09-10T21:14:57.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5199b9c9-b511-43de-bc53-6987d1c26826 -i td/osd-scrub-repair/2/new.json 2024-09-10T21:14:57.837 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:14:57.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:14:57.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRtuBmF84pGhAAGXPSuD45CDqTlqBtRtg5/w== --osd-uuid 5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:14:57.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:57.874+0000 7f4cc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:57.930 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:57.926+0000 7f4cc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:57.927+0000 7f4cc841e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:14:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:57.927+0000 7f4cc841e640 -1 bdev(0x557d64391c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:14:57.931 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:14:57.927+0000 7f4cc841e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:15:03.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:15:03.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:15:03.803 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:15:03.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:15:03.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:15:04.180 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:15:04.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:15:04.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:15:04.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:15:04.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:15:04.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:15:04.208 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:04.204+0000 7eff7a01d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:04.210 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:04.206+0000 7eff7a01d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:04.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:04.207+0000 7eff7a01d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:15:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:04.455 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:15:04.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:15:04.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:04.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:04.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:05.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:05.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:05.729 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:15:05.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:15:05.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:05.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:05.807 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:05.803+0000 7eff7a01d640 -1 Falling back to public interface 2024-09-10T21:15:06.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:07.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:07.018 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:15:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:15:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:07.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:07.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:07.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:07.716+0000 7eff7a01d640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:15:08.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:08.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:08.311 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:15:08.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:15:08.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:08.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:08.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:09.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:09.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:09.731 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:15:09.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:15:09.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:09.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:10.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:11.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:11.049 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:15:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:15:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:11.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:11.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:12.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:12.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:12.369 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:15:12.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:15:12.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:12.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:12.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:13.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:13.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:13.763 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:15:13.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:15:13.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:13.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:14.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:15.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:15.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:15.054 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:15:15.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:15:15.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:15.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:15.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:16.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:16.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:16.360 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:15:16.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:15:16.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:16.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:16.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:17.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:17.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:17.763 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:15:17.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:15:17.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:17.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:18.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:19.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:19.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:19.052 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:15:19.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:15:19.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:19.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:19.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:20.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:20.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:20.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:15:20.369 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:15:20.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:20.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:20.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:21.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:21.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:21.909 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:15:21.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:15:21.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:21.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:22.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:23.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:23.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:23.618 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:15:23.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:15:23.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:23.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:24.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:25.521 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:15:25.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:15:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:25.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:25.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:26.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:26.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:26.837 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:15:26.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:15:26.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:26.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:27.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:28.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:28.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:28.282 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:15:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:15:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:28.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:28.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:29.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:29.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:29.803 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:15:29.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:15:29.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:29.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:30.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:31.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:31.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:31.594 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:15:31.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:15:31.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:31.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:15:31.883 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1192235451,v1:127.0.0.1:6819/1192235451] [v2:127.0.0.1:6820/1192235451,v1:127.0.0.1:6821/1192235451] exists,up 5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:15:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:15:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:15:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:15:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-10T21:15:31.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:15:32.171 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:15:32.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:15:32.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:15:32.623 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:15:32.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:15:33.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:15:35.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-10T21:15:35.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-10T21:15:35.859 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' created 2024-09-10T21:15:35.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:15:36.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-10T21:15:37.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:15:37.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:15:38.067 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:15:38.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:15:39.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T21:15:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:15:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:15:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:15:39.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:15:39.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:15:39.083 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:15:39.083 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:15:39.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:15:39.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:15:39.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:15:39.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:15:39.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:15:39.465 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:15:39.465 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:15:39.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:15:39.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:39.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:15:39.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T21:15:39.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T21:15:39.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T21:15:39.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:39.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:15:39.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-10T21:15:39.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-10T21:15:39.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974' 2024-09-10T21:15:39.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:39.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:15:39.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974 2-64424509446' 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:39.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:15:39.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T21:15:39.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T21:15:39.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T21:15:39.783 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T21:15:39.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:40.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-10T21:15:40.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:40.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-10T21:15:40.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:40.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:15:40.063 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-10T21:15:40.063 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:40.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-10T21:15:40.064 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672974 2024-09-10T21:15:40.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-10T21:15:40.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:15:40.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-10T21:15:40.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:40.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-10T21:15:40.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:40.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:15:40.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-10T21:15:40.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:40.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-10T21:15:40.384 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509446 2024-09-10T21:15:40.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-10T21:15:40.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:15:40.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-10T21:15:40.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:15:40.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:15:40.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:15:41.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:15:41.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:15:41.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:15:41.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:15:41.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:15:41.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:15:41.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:15:41.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:15:41.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:15:41.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:15:41.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:15:41.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:15:41.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:15:41.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:15:41.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:15:41.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:15:41.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:15:41.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:15:41.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:15:41.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:15:41.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:15:41.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:15:42.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:15:42.013 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:15:42.013 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:15:42.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:15:42.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:42.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:15:42.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-10T21:15:42.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-10T21:15:42.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-10T21:15:42.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:42.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:15:42.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-10T21:15:42.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-10T21:15:42.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-09-10T21:15:42.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:42.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:15:42.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T21:15:42.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T21:15:42.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509448' 2024-09-10T21:15:42.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:42.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-10T21:15:42.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:42.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:15:42.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-10T21:15:42.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:42.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-10T21:15:42.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-10T21:15:42.400 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836502 2024-09-10T21:15:42.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:42.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-10T21:15:42.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:15:43.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:15:43.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:43.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-10T21:15:43.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:15:44.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:15:44.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:45.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-10T21:15:45.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:45.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-10T21:15:45.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:45.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:15:45.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-10T21:15:45.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:45.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-10T21:15:45.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-10T21:15:45.266 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672975 2024-09-10T21:15:45.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:15:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-10T21:15:45.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:45.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T21:15:45.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:45.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:15:45.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T21:15:45.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T21:15:45.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T21:15:45.547 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509448 2024-09-10T21:15:45.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:15:45.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-10T21:15:45.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:15:45.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:15:45.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:15:46.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:15:46.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:15:46.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:15:46.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:15:46.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:15:46.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:15:46.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:15:46.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:15:46.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:15:46.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:15:46.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:15:47.314 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:15:47.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:15:47.892 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:15:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:15:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:15:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:15:48.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:15:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:15:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:15:48.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:15:48.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:15:48.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:15:48.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:15:48.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:15:48.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:15:48.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:15:48.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:15:50.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:15:50.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:15:50.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:15:50.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:15:50.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:15:50.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:15:50.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:15:50.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:15:50.802 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:15:50.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:15:50.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:15:50.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:15:50.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:15:50.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:15:50.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:15:50.831 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:50.827+0000 7f9bb1ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:50.833 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:50.829+0000 7f9bb1ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:50.834 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:50.830+0000 7f9bb1ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:15:51.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:51.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:15:51.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:52.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:52.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:52.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:15:52.361 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:15:52.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:52.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:15:52.447 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:52.443+0000 7f9bb1ba4640 -1 Falling back to public interface 2024-09-10T21:15:52.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:53.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:15:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:15:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:53.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:15:53.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:15:54.137 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:15:54.133+0000 7f9bb1ba4640 -1 osd.1 34 log_to_monitors true 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:15:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:15:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/3510599707,v1:127.0.0.1:6811/3510599707] [v2:127.0.0.1:6812/3510599707,v1:127.0.0.1:6813/3510599707] exists,up 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:15:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:15:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:15:55.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:15:55.323 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:15:55.324 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:15:55.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:15:55.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:15:55.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:15:55.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:15:55.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:15:55.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:15:55.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:15:55.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:15:55.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:15:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:15:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:15:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:15:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:15:55.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:55.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:15:55.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T21:15:55.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T21:15:55.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T21:15:55.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:55.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:15:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-10T21:15:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-10T21:15:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-163208757250' 2024-09-10T21:15:56.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:15:56.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:15:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T21:15:56.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T21:15:56.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-163208757250 2-64424509451' 2024-09-10T21:15:56.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:56.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T21:15:56.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:56.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:15:56.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T21:15:56.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:56.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T21:15:56.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T21:15:56.204 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836506 2024-09-10T21:15:56.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:56.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-10T21:15:56.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:15:57.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:15:57.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:57.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-10T21:15:57.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:15:58.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:15:58.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:15:59.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T21:15:59.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:59.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757250 2024-09-10T21:15:59.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:59.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:15:59.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757250 2024-09-10T21:15:59.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:59.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-10T21:15:59.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-10T21:15:59.096 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 163208757250 2024-09-10T21:15:59.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:15:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-10T21:15:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:15:59.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T21:15:59.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:15:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:15:59.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T21:15:59.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:15:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T21:15:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T21:15:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509451 2024-09-10T21:15:59.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:15:59.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-10T21:15:59.648 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:15:59.648 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:15:59.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:15:59.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:15:59.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:15:59.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:00.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:16:00.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:00.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:00.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:16:00.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:16:01.031 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:16:01.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:16:01.532 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:16:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:16:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:16:01.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:16:01.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-10T21:16:01.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:16:01.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:16:01.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-10T21:16:01.730 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1366e-05 s, 33.4 MB/s 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:16:01.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:16:01.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:16:02.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:16:02.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:16:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:16:04.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:16:04.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:16:04.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:16:04.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:16:04.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:16:04.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:16:04.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:04.450+0000 7f04f94dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:04.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:04.453+0000 7f04f94dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:04.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:04.454+0000 7f04f94dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:04.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:16:04.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:16:04.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:16:04.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:16:04.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:16:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:16:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:16:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:04.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:04.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:05.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:05.806+0000 7f04f94dc640 -1 Falling back to public interface 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:05.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:06.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:07.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:07.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:07.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:16:07.294 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:16:07.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:07.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:07.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:07.401+0000 7f04f94dc640 -1 osd.0 42 log_to_monitors true 2024-09-10T21:16:07.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:08.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:08.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:08.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:16:08.590 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:16:08.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:08.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 43 last_clean_interval [5,42) [v2:127.0.0.1:6802/2537553299,v1:127.0.0.1:6803/2537553299] [v2:127.0.0.1:6804/2537553299,v1:127.0.0.1:6805/2537553299] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:16:08.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:16:08.956 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:16:08.956 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:16:08.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:16:08.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:16:08.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:16:09.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:16:09.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:16:09.337 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:09.337 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:16:09.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:16:09.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:09.338 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:16:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462914 2024-09-10T21:16:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462914 2024-09-10T21:16:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914' 2024-09-10T21:16:09.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:09.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:16:09.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757254 2024-09-10T21:16:09.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757254 2024-09-10T21:16:09.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914 1-163208757254' 2024-09-10T21:16:09.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:09.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:16:09.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-10T21:16:09.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-10T21:16:09.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914 1-163208757254 2-64424509455' 2024-09-10T21:16:09.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:09.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-201863462914 2024-09-10T21:16:09.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:09.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:16:09.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-201863462914 2024-09-10T21:16:09.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:09.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462914 2024-09-10T21:16:09.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2024-09-10T21:16:09.713 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 201863462914 2024-09-10T21:16:09.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462914 -lt 201863462914 2024-09-10T21:16:09.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:09.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757254 2024-09-10T21:16:09.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:09.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:16:09.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757254 2024-09-10T21:16:09.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:09.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757254 2024-09-10T21:16:09.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757254' 2024-09-10T21:16:09.990 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 163208757254 2024-09-10T21:16:09.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:16:10.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757254 -lt 163208757254 2024-09-10T21:16:10.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:10.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-10T21:16:10.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:10.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:16:10.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-10T21:16:10.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:10.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-10T21:16:10.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-10T21:16:10.283 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509455 2024-09-10T21:16:10.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:16:10.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-10T21:16:10.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:16:10.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:10.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:10.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:16:10.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:10.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:16:11.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:11.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:11.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:11.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:16:11.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:16:11.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:16:11.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:16:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:16:11.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:16:12.411 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:16:12.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:16:12.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:16:12.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:16:12.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:16:12.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:16:12.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:16:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-10T21:16:14.874 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:16:15.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:16:15.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:16:15.401 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:16:15.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:16:15.402 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:16:15.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:16:15.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:16:15.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:16:15.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:16:15.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:15.426+0000 7f178f6d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:15.433 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:15.428+0000 7f178f6d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:15.434 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:15.429+0000 7f178f6d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:16:15.686 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:16:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:16:15.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:16.325 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:16.320+0000 7f178f6d6640 -1 Falling back to public interface 2024-09-10T21:16:16.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:16.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:16.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:16:16.972 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:16.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:16.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:16:17.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:17.697+0000 7f178f6d6640 -1 osd.1 52 log_to_monitors true 2024-09-10T21:16:18.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:16:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:16:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:18.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:16:18.538 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [38,52) [v2:127.0.0.1:6810/2184287053,v1:127.0.0.1:6811/2184287053] [v2:127.0.0.1:6812/2184287053,v1:127.0.0.1:6813/2184287053] exists,up 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:16:18.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:16:18.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:16:18.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:16:18.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:16:18.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:16:18.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:16:18.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:16:18.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:16:18.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:16:18.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:19.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:16:19.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462917 2024-09-10T21:16:19.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462917 2024-09-10T21:16:19.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462917' 2024-09-10T21:16:19.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:19.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:16:19.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-10T21:16:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-10T21:16:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462917 1-244813135874' 2024-09-10T21:16:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:19.475 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:16:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T21:16:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T21:16:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462917 1-244813135874 2-64424509458' 2024-09-10T21:16:19.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:19.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-201863462917 2024-09-10T21:16:19.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:19.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:16:19.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:19.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-201863462917 2024-09-10T21:16:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462917 2024-09-10T21:16:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 201863462917' 2024-09-10T21:16:19.584 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 201863462917 2024-09-10T21:16:19.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:19.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462916 -lt 201863462917 2024-09-10T21:16:19.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:16:20.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:16:20.897 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:21.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462917 -lt 201863462917 2024-09-10T21:16:21.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:21.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-10T21:16:21.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:21.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:16:21.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-10T21:16:21.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:21.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-10T21:16:21.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-10T21:16:21.188 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135874 2024-09-10T21:16:21.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:16:21.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-10T21:16:21.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:21.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T21:16:21.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:21.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:16:21.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T21:16:21.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:21.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T21:16:21.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T21:16:21.468 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509458 2024-09-10T21:16:21.468 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:16:21.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-10T21:16:21.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:16:21.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:21.742 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:22.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:22.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:16:22.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:22.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:22.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:22.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:16:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-10T21:16:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:16:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-10T21:16:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:16:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:16:22.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:16:22.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:22.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:22.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:22.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:22.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:22.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:22.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:23.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:16:23.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:23.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:23.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:16:23.445 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:16:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:16:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:16:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:16:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:16:23.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:16:23.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:16:23.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:23.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:16:24.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:24.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:24.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-10T21:16:24.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:16:24.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:16:24.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-10T21:16:24.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:16:24.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:16:24.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:16:24.988 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:16:24.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:16:25.498 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:16:25.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:16:25.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:16:25.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:16:25.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-10T21:16:25.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:16:25.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:16:25.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-10T21:16:25.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-10T21:16:26.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-10T21:16:26.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:16:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:16:26.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:16:26.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:16:26.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:16:26.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:16:28.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:16:28.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:16:28.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:16:28.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:16:28.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:16:28.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:16:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:16:28.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:16:28.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:16:28.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:16:28.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:16:28.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:16:28.973 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:28.969+0000 7fda9ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:28.975 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:28.971+0000 7fda9ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:28.977 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:28.973+0000 7fda9ddcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:29.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:30.335 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:30.331+0000 7fda9ddcc640 -1 Falling back to public interface 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:30.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:30.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:31.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:31.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:31.893+0000 7fda9ddcc640 -1 osd.0 61 log_to_monitors true 2024-09-10T21:16:32.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:33.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:33.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:33.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:16:33.171 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:16:33.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:33.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:16:33.572 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [47,61) [v2:127.0.0.1:6802/4011214064,v1:127.0.0.1:6803/4011214064] [v2:127.0.0.1:6804/4011214064,v1:127.0.0.1:6805/4011214064] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:16:33.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:16:33.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:16:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:16:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:16:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:16:33.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:16:33.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:16:33.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:16:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:16:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:16:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:16:33.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:33.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:16:34.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841538 2024-09-10T21:16:34.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841538 2024-09-10T21:16:34.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538' 2024-09-10T21:16:34.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:34.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:16:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135878 2024-09-10T21:16:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135878 2024-09-10T21:16:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538 1-244813135878' 2024-09-10T21:16:34.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:34.241 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:16:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-10T21:16:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-10T21:16:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538 1-244813135878 2-64424509462' 2024-09-10T21:16:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:34.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841538 2024-09-10T21:16:34.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:34.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:16:34.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841538 2024-09-10T21:16:34.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:34.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841538 2024-09-10T21:16:34.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-09-10T21:16:34.348 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 283467841538 2024-09-10T21:16:34.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:34.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 283467841538 2024-09-10T21:16:34.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:16:35.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:16:35.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:35.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 283467841538 2024-09-10T21:16:35.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:16:36.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:16:36.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:37.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841538 -lt 283467841538 2024-09-10T21:16:37.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:37.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135878 2024-09-10T21:16:37.193 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:37.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:16:37.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135878 2024-09-10T21:16:37.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:37.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135878 2024-09-10T21:16:37.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-09-10T21:16:37.196 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135878 2024-09-10T21:16:37.196 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:16:37.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135878 -lt 244813135878 2024-09-10T21:16:37.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:37.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-10T21:16:37.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:37.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:16:37.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-10T21:16:37.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-10T21:16:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-10T21:16:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509462 2024-09-10T21:16:37.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:16:37.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-10T21:16:37.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:16:37.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:37.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:38.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:16:38.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:38.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:38.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:38.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:38.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:38.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:38.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:38.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:16:38.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:38.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:38.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:38.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:16:38.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:16:38.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:16:39.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:16:39.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:16:39.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:16:39.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:16:39.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:16:39.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:16:39.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:16:39.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:16:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:41.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:16:41.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:16:41.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:16:41.369 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:16:41.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:16:41.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:16:41.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:16:41.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:16:41.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:16:41.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:16:41.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:41.395+0000 7fcc3f2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:41.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:41.398+0000 7fcc3f2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:41.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:41.399+0000 7fcc3f2b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:41.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:41.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:42.755 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:42.751+0000 7fcc3f2b6640 -1 Falling back to public interface 2024-09-10T21:16:42.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:42.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:42.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:16:42.929 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:42.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:42.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:43.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:44.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:44.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:44.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:16:44.217 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:16:44.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:44.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:44.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:44.954+0000 7fcc3f2b6640 -1 osd.2 69 log_to_monitors true 2024-09-10T21:16:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:16:45.514 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:16:45.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:45.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1411131049,v1:127.0.0.1:6819/1411131049] [v2:127.0.0.1:6820/1411131049,v1:127.0.0.1:6821/1411131049] exists,up 5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:16:45.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:16:45.827 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:16:45.827 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:16:45.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:16:45.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:16:45.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:16:45.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:16:45.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:16:45.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:16:45.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:16:45.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:16:45.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:46.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:16:46.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841541 2024-09-10T21:16:46.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841541 2024-09-10T21:16:46.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841541' 2024-09-10T21:16:46.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:46.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:16:46.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135881 2024-09-10T21:16:46.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135881 2024-09-10T21:16:46.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841541 1-244813135881' 2024-09-10T21:16:46.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:46.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:16:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-10T21:16:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-10T21:16:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841541 1-244813135881 2-317827579906' 2024-09-10T21:16:46.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:46.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841541 2024-09-10T21:16:46.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:46.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:16:46.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841541 2024-09-10T21:16:46.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:46.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841541 2024-09-10T21:16:46.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-09-10T21:16:46.782 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 283467841541 2024-09-10T21:16:46.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:47.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841540 -lt 283467841541 2024-09-10T21:16:47.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:16:48.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:16:48.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:48.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841542 -lt 283467841541 2024-09-10T21:16:48.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:48.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135881 2024-09-10T21:16:48.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:48.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:16:48.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135881 2024-09-10T21:16:48.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:48.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135881 2024-09-10T21:16:48.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135881' 2024-09-10T21:16:48.356 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135881 2024-09-10T21:16:48.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:16:48.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135881 -lt 244813135881 2024-09-10T21:16:48.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:48.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-317827579906 2024-09-10T21:16:48.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:48.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:16:48.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-317827579906 2024-09-10T21:16:48.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-10T21:16:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-09-10T21:16:48.635 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 317827579906 2024-09-10T21:16:48.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:16:48.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-10T21:16:48.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:16:48.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:48.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:49.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:49.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:16:49.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:49.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:49.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:16:49.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:16:49.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:16:49.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:16:49.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:16:50.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:16:50.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:16:50.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:16:50.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:16:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:16:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:16:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:16:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-10T21:16:50.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:16:50.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:16:50.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:16:50.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:16:50.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:16:50.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:16:53.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:16:53.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:16:53.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:16:53.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:16:53.315 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:16:53.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:16:53.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:16:53.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:16:53.339 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:53.335+0000 7f70c8dc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:53.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:53.338+0000 7f70c8dc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:53.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:53.339+0000 7f70c8dc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:53.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:53.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:54.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:54.454+0000 7f70c8dc3640 -1 Falling back to public interface 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:54.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:55.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:56.117 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:16:56.113+0000 7f70c8dc3640 -1 osd.2 77 log_to_monitors true 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:56.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:56.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:16:57.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:16:57.828 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/1825892363,v1:127.0.0.1:6819/1825892363] [v2:127.0.0.1:6820/1825892363,v1:127.0.0.1:6821/1825892363] exists,up 5199b9c9-b511-43de-bc53-6987d1c26826 2024-09-10T21:16:57.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:16:57.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:16:57.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:16:57.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:16:57.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:16:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:16:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:16:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:16:58.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:16:58.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:58.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:16:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841545 2024-09-10T21:16:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841545 2024-09-10T21:16:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841545' 2024-09-10T21:16:58.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:58.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:16:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135885 2024-09-10T21:16:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135885 2024-09-10T21:16:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841545 1-244813135885' 2024-09-10T21:16:58.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:16:58.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:16:58.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-10T21:16:58.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-10T21:16:58.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841545 1-244813135885 2-352187318274' 2024-09-10T21:16:58.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:16:58.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841545 2024-09-10T21:16:58.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:16:58.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:16:58.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841545 2024-09-10T21:16:58.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:16:58.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841545 2024-09-10T21:16:58.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-09-10T21:16:58.513 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 283467841545 2024-09-10T21:16:58.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:16:58.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841544 -lt 283467841545 2024-09-10T21:16:58.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:16:59.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:16:59.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:00.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841545 -lt 283467841545 2024-09-10T21:17:00.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:00.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135885 2024-09-10T21:17:00.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:00.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:17:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135885 2024-09-10T21:17:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135885 2024-09-10T21:17:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135885' 2024-09-10T21:17:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 244813135885 2024-09-10T21:17:00.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:17:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135885 -lt 244813135885 2024-09-10T21:17:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:00.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-10T21:17:00.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:00.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:17:00.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-10T21:17:00.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-10T21:17:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-10T21:17:00.362 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318274 2024-09-10T21:17:00.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:17:00.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-10T21:17:00.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:17:00.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:00.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:00.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:00.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:00.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:01.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:17:01.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:01.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:01.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:17:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:17:01.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:17:01.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-10T21:17:01.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:17:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:17:01.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:17:01.982 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:17:01.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:17:02.544 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:17:02.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:17:02.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:17:02.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:17:02.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-10T21:17:02.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:17:02.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:17:02.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records in 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records out 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.9595e-05 s, 58.9 MB/s 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:17:02.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:17:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:17:03.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:17:03.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:17:03.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:17:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:17:05.367 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:17:05.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:17:05.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:17:05.369 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:17:05.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:17:05.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:17:05.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:17:05.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:17:05.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:17:05.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:17:05.397 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:05.392+0000 7f16b920b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:05.400 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:05.395+0000 7f16b920b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:05.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:05.396+0000 7f16b920b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:05.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:17:05.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:17:05.655 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:17:05.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:05.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:05.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:17:06.936 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:06.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:06.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:07.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:07.267 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:07.262+0000 7f16b920b640 -1 Falling back to public interface 2024-09-10T21:17:08.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:08.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:08.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:17:08.223 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:17:08.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:08.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:08.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:08.951+0000 7f16b920b640 -1 osd.1 86 log_to_monitors true 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:09.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:09.872 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 91 up_thru 82 down_at 87 last_clean_interval [57,86) [v2:127.0.0.1:6810/223737332,v1:127.0.0.1:6811/223737332] [v2:127.0.0.1:6812/223737332,v1:127.0.0.1:6813/223737332] exists,up 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:17:09.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:17:09.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:17:09.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:17:09.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:17:09.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:17:09.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:17:09.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:17:10.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:17:10.282 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:10.282 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:17:10.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:17:10.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:10.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:17:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841548 2024-09-10T21:17:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841548 2024-09-10T21:17:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841548' 2024-09-10T21:17:10.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:10.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:17:10.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-10T21:17:10.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-10T21:17:10.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841548 1-390842023938' 2024-09-10T21:17:10.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:10.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:17:10.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318277 2024-09-10T21:17:10.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318277 2024-09-10T21:17:10.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841548 1-390842023938 2-352187318277' 2024-09-10T21:17:10.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:10.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841548 2024-09-10T21:17:10.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:10.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:17:10.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841548 2024-09-10T21:17:10.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:10.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841548 2024-09-10T21:17:10.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-09-10T21:17:10.736 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 283467841548 2024-09-10T21:17:10.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:11.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841547 -lt 283467841548 2024-09-10T21:17:11.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:17:12.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:17:12.069 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:12.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841548 -lt 283467841548 2024-09-10T21:17:12.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:12.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023938 2024-09-10T21:17:12.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:12.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:17:12.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023938 2024-09-10T21:17:12.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:12.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-10T21:17:12.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-09-10T21:17:12.357 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 390842023938 2024-09-10T21:17:12.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:17:12.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-10T21:17:12.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:12.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318277 2024-09-10T21:17:12.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:12.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:17:12.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318277 2024-09-10T21:17:12.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:12.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318277 2024-09-10T21:17:12.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318277' 2024-09-10T21:17:12.637 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318277 2024-09-10T21:17:12.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:17:13.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318278 -lt 352187318277 2024-09-10T21:17:13.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:17:13.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:13.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:13.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:17:13.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:13.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:13.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:13.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:13.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:13.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:13.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:13.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:17:13.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:13.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:13.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:14.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:17:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T21:17:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:17:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T21:17:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:17:14.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:17:14.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:17:14.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:14.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:14.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:17:14.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:14.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:14.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:17:14.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:17:14.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:17:14.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:17:15.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:17:15.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:17:15.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:17:15.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:17:15.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:17:15.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:17:15.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:15.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:15.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:17:15.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:15.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:15.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:17:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T21:17:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:17:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:17:15.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:17:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:17:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:17:15.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:17:15.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:17:16.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:17:16.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:17:16.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T21:17:16.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:17:16.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T21:17:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:17:16.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:16.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:16.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:17:16.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:16.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:16.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:17:17.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:17:17.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:17:17.667 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:17:17.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:17:18.095 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:17:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:17:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:17:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:17:18.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:17:18.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:17:18.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:17:18.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:18.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:17:21.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:17:21.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:17:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:17:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:17:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:17:21.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:17:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:17:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:17:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:17:21.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:17:21.082 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:21.078+0000 7f4eafced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:21.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:21.081+0000 7f4eafced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:21.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:21.082+0000 7f4eafced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:17:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:21.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:21.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:22.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:22.477+0000 7f4eafced640 -1 Falling back to public interface 2024-09-10T21:17:22.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:22.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:22.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:17:22.618 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:22.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:22.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:22.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:23.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:23.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:23.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:17:23.907 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:17:23.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:23.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:24.111+0000 7f4eafced640 -1 osd.0 95 log_to_monitors true 2024-09-10T21:17:24.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:25.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [66,95) [v2:127.0.0.1:6802/3616241912,v1:127.0.0.1:6803/3616241912] [v2:127.0.0.1:6804/3616241912,v1:127.0.0.1:6805/3616241912] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:17:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:17:25.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:17:25.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:17:25.797 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:17:25.797 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:17:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:17:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:17:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:17:25.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:26.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:17:26.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-10T21:17:26.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-10T21:17:26.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602' 2024-09-10T21:17:26.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:26.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:17:26.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023942 2024-09-10T21:17:26.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023942 2024-09-10T21:17:26.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942' 2024-09-10T21:17:26.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:26.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:17:26.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318282 2024-09-10T21:17:26.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318282 2024-09-10T21:17:26.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942 2-352187318282' 2024-09-10T21:17:26.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:26.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729602 2024-09-10T21:17:26.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:26.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:17:26.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729602 2024-09-10T21:17:26.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:26.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-10T21:17:26.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-09-10T21:17:26.539 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 429496729602 2024-09-10T21:17:26.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:26.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 429496729602 2024-09-10T21:17:26.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:17:27.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:17:27.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:28.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729602 -lt 429496729602 2024-09-10T21:17:28.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:28.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023942 2024-09-10T21:17:28.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:28.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:17:28.105 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023942 2024-09-10T21:17:28.105 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:28.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023942 2024-09-10T21:17:28.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023942' 2024-09-10T21:17:28.107 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 390842023942 2024-09-10T21:17:28.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:17:28.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023942 -lt 390842023942 2024-09-10T21:17:28.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:28.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318282 2024-09-10T21:17:28.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:28.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:17:28.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318282 2024-09-10T21:17:28.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:28.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318282 2024-09-10T21:17:28.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2024-09-10T21:17:28.383 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318282 2024-09-10T21:17:28.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:17:28.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318282 -lt 352187318282 2024-09-10T21:17:28.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:17:28.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:28.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:29.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:17:29.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:29.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:29.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:17:29.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:29.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:29.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:17:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:17:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:17:29.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:17:29.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:17:29.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:17:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:17:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:17:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:17:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:17:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:17:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:17:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:17:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:17:29.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:17:29.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:17:29.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:17:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:17:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:17:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:17:32.097 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:17:32.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:17:32.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:17:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:17:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:17:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:17:32.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:17:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:32.122+0000 7f7f25015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:32.128 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:32.124+0000 7f7f25015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:32.129 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:32.125+0000 7f7f25015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:32.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:32.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:32.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:33.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:33.248+0000 7f7f25015640 -1 Falling back to public interface 2024-09-10T21:17:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:33.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:17:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:33.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:33.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:34.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:34.878+0000 7f7f25015640 -1 osd.1 102 log_to_monitors true 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:34.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:35.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:36.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:17:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:17:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:36.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/2895761966,v1:127.0.0.1:6811/2895761966] [v2:127.0.0.1:6812/2895761966,v1:127.0.0.1:6813/2895761966] exists,up 1fad0485-4110-4f5e-8314-153221e152ff 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:17:36.648 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:17:36.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:17:36.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:17:36.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:17:36.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:17:36.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:17:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:17:37.016 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:17:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:17:37.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:37.017 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:17:37.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729605 2024-09-10T21:17:37.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729605 2024-09-10T21:17:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605' 2024-09-10T21:17:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:37.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:17:37.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-10T21:17:37.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-10T21:17:37.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-09-10T21:17:37.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:37.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:17:37.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318285 2024-09-10T21:17:37.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318285 2024-09-10T21:17:37.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-352187318285' 2024-09-10T21:17:37.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:37.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729605 2024-09-10T21:17:37.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:37.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:17:37.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729605 2024-09-10T21:17:37.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:37.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729605 2024-09-10T21:17:37.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-09-10T21:17:37.328 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 429496729605 2024-09-10T21:17:37.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:37.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729604 -lt 429496729605 2024-09-10T21:17:37.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:17:38.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:17:38.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:38.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729605 -lt 429496729605 2024-09-10T21:17:38.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500674 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500674 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500674 2024-09-10T21:17:38.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:17:39.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-10T21:17:39.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:39.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318285 2024-09-10T21:17:39.192 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:39.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:17:39.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318285 2024-09-10T21:17:39.194 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:39.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318285 2024-09-10T21:17:39.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-09-10T21:17:39.195 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318285 2024-09-10T21:17:39.195 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:17:39.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318285 -lt 352187318285 2024-09-10T21:17:39.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:17:39.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:39.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:39.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:17:39.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:39.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:39.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:39.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:39.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:39.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:39.819 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:40.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:17:40.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:40.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:40.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:17:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:17:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-10T21:17:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:17:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:17:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:17:40.841 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:17:40.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:17:41.335 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:17:41.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:17:41.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:17:41.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:17:41.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-10T21:17:41.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:17:41.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:17:41.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-10T21:17:41.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-10T21:17:41.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:17:41.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:17:41.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:42.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-10T21:17:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:17:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:17:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:17:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:17:44.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:17:44.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:17:44.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:17:44.097 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:17:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:17:44.120 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:44.116+0000 7fe1ffbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:44.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:44.128+0000 7fe1ffbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:44.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:44.129+0000 7fe1ffbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:44.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:17:44.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:17:44.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:17:44.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:44.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:44.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:45.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:45.231+0000 7fe1ffbdb640 -1 Falling back to public interface 2024-09-10T21:17:45.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:17:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:45.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:46.553 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:46.549+0000 7fe1ffbdb640 -1 osd.0 111 log_to_monitors true 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:46.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:47.377 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 116 up_thru 107 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/1348546748,v1:127.0.0.1:6803/1348546748] [v2:127.0.0.1:6804/1348546748,v1:127.0.0.1:6805/1348546748] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:17:47.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:17:47.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:17:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:17:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:17:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:17:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:17:47.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:47.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:17:48.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=498216206338 2024-09-10T21:17:48.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 498216206338 2024-09-10T21:17:48.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338' 2024-09-10T21:17:48.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:48.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:17:48.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500677 2024-09-10T21:17:48.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500677 2024-09-10T21:17:48.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677' 2024-09-10T21:17:48.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:48.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318288 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318288 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677 2-352187318288' 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-498216206338 2024-09-10T21:17:48.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:48.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:17:48.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-498216206338 2024-09-10T21:17:48.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:48.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=498216206338 2024-09-10T21:17:48.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-09-10T21:17:48.410 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 498216206338 2024-09-10T21:17:48.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:48.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 498216206338 2024-09-10T21:17:48.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:17:49.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:17:49.686 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:49.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 498216206338 2024-09-10T21:17:49.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:17:50.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:17:50.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:51.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-10T21:17:51.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:51.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500677 2024-09-10T21:17:51.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:17:51.269 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500677 2024-09-10T21:17:51.269 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:51.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500677 2024-09-10T21:17:51.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500677' 2024-09-10T21:17:51.270 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500677 2024-09-10T21:17:51.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:17:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500677 -lt 459561500677 2024-09-10T21:17:51.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318288 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318288 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:51.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318288 2024-09-10T21:17:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318288' 2024-09-10T21:17:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318288 2024-09-10T21:17:51.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:17:51.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318288 -lt 352187318288 2024-09-10T21:17:51.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:17:51.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:51.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:17:52.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:17:52.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:17:52.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:17:52.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:17:52.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:17:52.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:17:52.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:17:52.814 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:17:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:17:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:17:52.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:53.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:17:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:17:55.432 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:17:55.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:17:55.434 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:17:55.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:17:55.435 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:17:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:17:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:17:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:17:55.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:17:55.461 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:55.456+0000 7efec760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:55.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:55.458+0000 7efec760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:55.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:55.460+0000 7efec760e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:55.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:17:55.719 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:17:55.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:55.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:56.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:56.343 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:56.338+0000 7efec760e640 -1 Falling back to public interface 2024-09-10T21:17:57.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:57.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:57.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:17:57.002 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:57.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:57.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:57.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:17:57.708 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:17:57.703+0000 7efec760e640 -1 osd.0 118 log_to_monitors true 2024-09-10T21:17:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:17:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:17:58.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:17:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:17:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:17:58.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:17:58.598 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 123 up_thru 116 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/1332969519,v1:127.0.0.1:6803/1332969519] [v2:127.0.0.1:6804/1332969519,v1:127.0.0.1:6805/1332969519] exists,up b0a4720a-7033-4624-854d-8b1ac2c4478b 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:17:58.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:17:58.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:17:58.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:17:58.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:17:58.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:17:58.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:17:58.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:17:58.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:17:58.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:17:58.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:17:58.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:17:58.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:58.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:17:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-10T21:17:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-10T21:17:59.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410' 2024-09-10T21:17:59.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:59.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:17:59.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500680 2024-09-10T21:17:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500680 2024-09-10T21:17:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500680' 2024-09-10T21:17:59.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:17:59.448 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318291 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318291 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500680 2-352187318291' 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-528280977410 2024-09-10T21:17:59.597 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:17:59.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:17:59.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-528280977410 2024-09-10T21:17:59.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:17:59.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-10T21:17:59.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-09-10T21:17:59.600 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 528280977410 2024-09-10T21:17:59.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:17:59.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 528280977410 2024-09-10T21:17:59.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:18:00.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:18:00.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:18:01.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-10T21:18:01.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:18:01.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500680 2024-09-10T21:18:01.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:18:01.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:18:01.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500680 2024-09-10T21:18:01.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:18:01.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500680 2024-09-10T21:18:01.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500680' 2024-09-10T21:18:01.277 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500680 2024-09-10T21:18:01.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:18:01.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500680 -lt 459561500680 2024-09-10T21:18:01.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:18:01.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318291 2024-09-10T21:18:01.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:18:01.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:18:01.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318291 2024-09-10T21:18:01.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:18:01.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318291 2024-09-10T21:18:01.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318291' 2024-09-10T21:18:01.555 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318291 2024-09-10T21:18:01.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:18:01.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318291 -lt 352187318291 2024-09-10T21:18:01.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:18:01.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:18:01.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:18:02.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:18:02.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:18:02.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:18:02.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:18:02.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:18:02.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:18:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-10T21:18:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:18:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-10T21:18:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:18:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:18:02.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:18:02.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:18:02.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:18:02.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:18:02.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:18:02.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:18:02.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:18:02.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:18:03.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:18:03.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:18:03.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:18:03.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:18:03.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:18:03.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:18:03.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:18:03.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:18:03.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:18:03.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:18:03.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:18:04.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:18:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:18:04.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:18:04.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:18:04.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:18:04.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:18:04.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:18:04.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:18:04.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-10T21:18:04.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-10T21:18:04.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:18:04.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-10T21:18:04.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-10T21:18:04.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:04.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:04.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:04.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:05.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-10T21:18:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-10T21:18:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:18:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:18:05.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:05.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:05.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:05.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:06.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:06.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:06.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:06.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:06.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:06.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:06.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:07.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:07.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:08.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:08.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:08.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:08.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:08.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:08.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:08.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:08.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:08.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:09.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:09.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:09.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:09.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:09.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:09.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:09.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:09.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:09.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:10.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:10.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:10.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:10.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:10.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:10.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:10.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:10.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:10.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:11.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:11.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:18:11.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:11.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:18:11.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:11.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:18:12.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:18:05.798088+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:12.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:18:12.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-10T21:18:12.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:18:12.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-10T21:18:12.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:18:12.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-10T21:18:12.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-10T21:18:12.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:18:12.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-09-10T21:18:12.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:18:12.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:18:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:18:12.300 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:18:12.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:18:12.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:18:12.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:18:12.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:18:12.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-10T21:18:12.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-10T21:18:12.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:12.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:18:12.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:12.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-10T21:18:12.854 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-10T21:15:37.946540+0000 last_deep_scrub_stamp 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:18:12.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:12.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:18:13.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:13.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:14.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:14.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:18:14.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:14.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:18:14.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:14.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:18:14.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:14.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:15.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:15.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:15.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:18:15.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:15.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:18:15.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:15.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:18:15.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:15:37.946540+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:15.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:18:16.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:18:16.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:18:16.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:18:16.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:18:16.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:18:16.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:18:16.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:18:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:18:13.027000+0000 '>' 2024-09-10T21:15:37.946540+0000 2024-09-10T21:18:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:18:17.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-10T21:18:17.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:18:17.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-10T21:18:17.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:18:17.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-10T21:18:17.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-10T21:18:17.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:18:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-09-10T21:18:17.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:18:17.133 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:18:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:18:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:18:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:18:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:18:17.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:18:17.156 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:18:17.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:18:17.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:18:17.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:18:17.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-10T21:18:17.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-10T21:18:17.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-10T21:18:17.550 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' removed 2024-09-10T21:18:17.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:18:17.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:18:17.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:18:17.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:18:17.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:18:17.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:18:17.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:18:17.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:18:17.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:18:17.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:18:17.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:18:17.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:18:17.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:18:17.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:18:17.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:18:17.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:18:17.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:18:17.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:18:17.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:18:17.675 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:18:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:18:17.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:18:17.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:18:17.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:18:17.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:17.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:17.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:18:17.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:18:17.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:18:17.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:18:17.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:18:17.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:18:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:18:17.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:18:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:18:17.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:18:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:18:17.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:18:17.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:18:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:18:17.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:18:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:18:17.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:18:17.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:18:17.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:18:17.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:18:17.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:18:17.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:17.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:17.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:18:17.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:18:17.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:18:17.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:18:17.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:18:17.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:17.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:17.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-10T21:18:17.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:18:17.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:18:18.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:18.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:18:18.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:18:18.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:18:18.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:18:18.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:18:18.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:18:18.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:18:18.361 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:18:18.361 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:18:18.361 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:18:18.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:18:18.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:18:18.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:18:18.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:19.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:19.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:18:19.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:18:19.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-10T21:18:19.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:18:19.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:18:19.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:18:19.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:18:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8d63ab54-734d-49f1-bdde-c448701618bc' 2024-09-10T21:18:19.367 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:18:19.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:18:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbt+BmbMh/FhAAg3BJyoOgRMh1+2ZP0hQKPA== 2024-09-10T21:18:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbt+BmbMh/FhAAg3BJyoOgRMh1+2ZP0hQKPA=="}' 2024-09-10T21:18:19.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d63ab54-734d-49f1-bdde-c448701618bc -i td/osd-scrub-repair/0/new.json 2024-09-10T21:18:19.584 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:18:19.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:18:19.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbt+BmbMh/FhAAg3BJyoOgRMh1+2ZP0hQKPA== --osd-uuid 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:18:19.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:19.621+0000 7ff3b3f69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:19.635 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:19.631+0000 7ff3b3f69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:19.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:19.632+0000 7ff3b3f69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:19.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:19.632+0000 7ff3b3f69640 -1 bdev(0x55948d7e2800 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:18:19.636 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:19.633+0000 7ff3b3f69640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:18:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:18:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:18:25.545 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:18:25.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:18:25.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:18:25.912 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:18:25.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:18:25.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:18:25.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:18:25.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:18:25.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:18:25.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:25.936+0000 7fcb089c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:25.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:25.938+0000 7fcb089c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:25.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:25.939+0000 7fcb089c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:18:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:18:26.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:26.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:26.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:27.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:27.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:27.469 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:18:27.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:18:27.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:27.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:27.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:27.563+0000 7fcb089c5640 -1 Falling back to public interface 2024-09-10T21:18:27.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:28.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:28.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:28.753 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:18:28.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:18:28.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:28.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:29.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:29.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:29.460+0000 7fcb089c5640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:18:30.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:30.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:30.042 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:18:30.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:18:30.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:30.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:30.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:31.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:31.410 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:18:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:18:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:31.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:31.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:32.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:32.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:32.695 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:18:32.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:18:32.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:32.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:33.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:34.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:34.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:34.061 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:18:34.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:18:34.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:34.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:34.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:35.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:35.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:35.473 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:18:35.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:18:35.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:35.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:35.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:36.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:36.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:36.757 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:18:36.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:18:36.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:36.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:37.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:38.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:38.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:38.043 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:18:38.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:18:38.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:38.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:38.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:39.449 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:18:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:18:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:39.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:39.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:40.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:40.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:40.738 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:18:40.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:18:40.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:40.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:41.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:42.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:42.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:42.166 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:18:42.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:18:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:42.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:42.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:43.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:43.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:43.815 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:18:43.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:18:43.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:43.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:44.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:45.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:45.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:45.220 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:18:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:18:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:45.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:46.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:46.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:46.747 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:18:46.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:18:46.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:46.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:47.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:48.301 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:18:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:18:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:48.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:49.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:49.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:49.961 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:18:49.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:18:49.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:49.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:50.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:51.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:51.709 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:18:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:18:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:51.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:52.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:18:53.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:18:53.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:18:53.286 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:18:53.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:18:53.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:18:53.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:18:53.594 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274911182,v1:127.0.0.1:6803/274911182] [v2:127.0.0.1:6804/274911182,v1:127.0.0.1:6805/274911182] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:18:53.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:18:53.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:18:53.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:18:53.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:18:53.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:18:53.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:18:53.601 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:18:53.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8' 2024-09-10T21:18:53.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:18:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9t+BmjWmEJBAAkNFcwuXR4YK13GCIbIG98A== 2024-09-10T21:18:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9t+BmjWmEJBAAkNFcwuXR4YK13GCIbIG98A=="}' 2024-09-10T21:18:53.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:18:53.965 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:18:53.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:18:53.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9t+BmjWmEJBAAkNFcwuXR4YK13GCIbIG98A== --osd-uuid 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:18:54.007 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:54.003+0000 7ff986f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:54.049 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:54.045+0000 7ff986f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:54.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:54.046+0000 7ff986f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:18:54.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:54.046+0000 7ff986f34640 -1 bdev(0x563141991c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:18:54.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:18:54.046+0000 7ff986f34640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:18:59.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:18:59.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:18:59.712 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:18:59.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:18:59.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:19:00.070 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:19:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:19:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:19:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:19:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:19:00.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:19:00.097 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:00.094+0000 7f2f6ea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:00.099 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:00.096+0000 7f2f6ea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:00.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:00.097+0000 7f2f6ea15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:19:00.347 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:19:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:00.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:00.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:01.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:01.200+0000 7f2f6ea15640 -1 Falling back to public interface 2024-09-10T21:19:01.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:01.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:01.623 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:19:01.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:19:01.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:01.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:01.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:02.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:02.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:02.912 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:19:02.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:19:02.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:02.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:03.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:03.049+0000 7f2f6ea15640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:19:03.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:04.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:04.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:04.263 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:19:04.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:19:04.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:04.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:04.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:05.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:05.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:05.553 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:19:05.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:19:05.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:05.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:05.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:06.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:06.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:06.883 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:19:06.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:19:06.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:06.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:08.172 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:19:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:19:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:08.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:09.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:09.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:09.459 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:19:09.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:19:09.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:09.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:09.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:10.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:10.757 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:19:10.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:19:10.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:10.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:11.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:12.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:12.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:12.045 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:19:12.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:19:12.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:12.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:12.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:13.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:13.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:13.381 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:19:13.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:19:13.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:13.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:14.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:14.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:14.755 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:19:14.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:19:14.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:14.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:15.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:16.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:16.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:16.225 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:19:16.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:19:16.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:16.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:17.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:17.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:17.929 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:19:17.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:19:17.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:17.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:18.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:19.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:19.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:19.270 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:19:19.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:19:19.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:19.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:19.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:20.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:20.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:20.605 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:19:20.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:19:20.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:20.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:21.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:22.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:22.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:22.333 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:19:22.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:19:22.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:22.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:22.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:23.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:23.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:23.820 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:19:23.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:19:23.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:23.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:24.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:25.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:25.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:25.300 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:19:25.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:19:25.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:25.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:19:25.610 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1963749300,v1:127.0.0.1:6811/1963749300] [v2:127.0.0.1:6812/1963749300,v1:127.0.0.1:6813/1963749300] exists,up 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:19:25.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:19:25.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:19:25.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:19:25.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:19:25.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:19:25.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:19:25.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:19:25.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:19:25.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38' 2024-09-10T21:19:25.617 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:19:25.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:19:25.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdt+BmBxt+JRAAUi87HLMPmKihSmxOQpHFig== 2024-09-10T21:19:25.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdt+BmBxt+JRAAUi87HLMPmKihSmxOQpHFig=="}' 2024-09-10T21:19:25.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 -i td/osd-scrub-repair/2/new.json 2024-09-10T21:19:26.096 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:19:26.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:19:26.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdt+BmBxt+JRAAUi87HLMPmKihSmxOQpHFig== --osd-uuid 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:19:26.138 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:26.135+0000 7f171020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:26.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:26.177+0000 7f171020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:26.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:26.178+0000 7f171020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:26.178+0000 7f171020d640 -1 bdev(0x55be6d86fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:19:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:26.178+0000 7f171020d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:19:32.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:19:32.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:19:32.409 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:19:32.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:19:32.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:19:32.770 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:19:32.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:19:32.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:19:32.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:19:32.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:19:32.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:19:32.797 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:32.794+0000 7fe6d8a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:32.799 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:32.796+0000 7fe6d8a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:32.797+0000 7fe6d8a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:33.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:19:33.050 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:19:33.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:33.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:33.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:34.153+0000 7fe6d8a0d640 -1 Falling back to public interface 2024-09-10T21:19:34.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:34.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:34.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:19:34.326 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:19:34.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:34.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:35.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:35.602 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:19:35.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:19:35.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:35.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:35.867 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:19:35.863+0000 7fe6d8a0d640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:19:35.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:36.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:36.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:36.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:19:36.892 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:19:36.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:36.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:37.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:38.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:38.179 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:19:38.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:38.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:19:38.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:38.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:38.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:39.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:39.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:39.502 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:19:39.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:19:39.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:39.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:39.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:40.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:40.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:40.789 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:19:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:19:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:40.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:41.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:42.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:42.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:42.076 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:19:42.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:19:42.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:42.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:42.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:43.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:43.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:43.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:19:43.363 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:19:43.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:43.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:43.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:44.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:44.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:44.703 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:19:44.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:19:44.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:44.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:44.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:45.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:45.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:45.995 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:19:45.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:19:45.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:45.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:46.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:47.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:47.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:47.282 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:19:47.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:19:47.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:47.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:47.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:48.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:48.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:48.937 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:19:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:19:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:49.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:50.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:50.453 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:19:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:19:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:50.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:51.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:51.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:51.973 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:19:51.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:19:51.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:51.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:52.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:53.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:53.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:53.449 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:19:53.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:19:53.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:53.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:54.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:55.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:55.066 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:19:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:19:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:55.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:55.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:56.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:56.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:56.594 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:19:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:19:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:56.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:57.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:19:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:19:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:19:58.166 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:19:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:19:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:19:58.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:19:58.466 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3825342303,v1:127.0.0.1:6819/3825342303] [v2:127.0.0.1:6820/3825342303,v1:127.0.0.1:6821/3825342303] exists,up 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:19:58.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:19:58.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:19:58.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:19:58.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-10T21:19:58.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:19:58.732 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:19:58.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:19:58.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:19:59.088 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:19:59.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:20:00.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:20:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-10T21:20:02.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-10T21:20:02.891 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' created 2024-09-10T21:20:02.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:20:03.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-10T21:20:04.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:20:04.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:20:05.176 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:20:05.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:20:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T21:20:06.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T21:20:06.656 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 3 allow_ec_overwrites to true 2024-09-10T21:20:06.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:20:06.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:20:06.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:20:06.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:20:06.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:20:06.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:20:06.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:20:06.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:20:06.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:20:06.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:20:06.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:20:07.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:20:07.071 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:07.071 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:20:07.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:20:07.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:07.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:20:07.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-10T21:20:07.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-10T21:20:07.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-10T21:20:07.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:07.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:20:07.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-10T21:20:07.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-10T21:20:07.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-10T21:20:07.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:07.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:20:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T21:20:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T21:20:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509448' 2024-09-10T21:20:07.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:07.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-10T21:20:07.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:07.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:20:07.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-10T21:20:07.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-10T21:20:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-10T21:20:07.490 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836500 2024-09-10T21:20:07.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:07.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-10T21:20:07.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:07.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-10T21:20:07.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:07.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:20:07.778 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-10T21:20:07.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:07.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-10T21:20:07.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-10T21:20:07.780 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672976 2024-09-10T21:20:07.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672976 2024-09-10T21:20:08.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:09.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:20:09.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:09.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672976 2024-09-10T21:20:09.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:10.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:20:10.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:10.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-10T21:20:10.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:10.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T21:20:10.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:10.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:20:10.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T21:20:10.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:10.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T21:20:10.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T21:20:10.650 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509448 2024-09-10T21:20:10.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:20:10.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-10T21:20:10.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:20:10.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:10.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:11.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:11.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:11.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:11.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:20:11.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:11.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:11.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:11.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:20:11.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:20:11.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:20:11.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:20:11.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:20:11.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:20:11.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:20:11.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:20:11.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:20:11.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:20:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:20:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:20:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:20:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:12.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:20:12.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-10T21:20:12.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-10T21:20:12.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-10T21:20:12.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:12.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:20:12.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-10T21:20:12.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-10T21:20:12.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672978' 2024-09-10T21:20:12.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:12.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:20:12.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-10T21:20:12.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-10T21:20:12.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672978 2-64424509450' 2024-09-10T21:20:12.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:12.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-10T21:20:12.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:12.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:20:12.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-10T21:20:12.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:12.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-10T21:20:12.577 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836502 2024-09-10T21:20:12.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-10T21:20:12.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:12.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-10T21:20:12.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:13.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:20:13.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:14.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836502 2024-09-10T21:20:14.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-10T21:20:14.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:14.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:20:14.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-10T21:20:14.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:14.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-10T21:20:14.142 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672978 2024-09-10T21:20:14.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-10T21:20:14.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:14.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-10T21:20:14.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:14.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-10T21:20:14.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:20:14.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-10T21:20:14.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:14.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-10T21:20:14.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-10T21:20:14.420 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509450 2024-09-10T21:20:14.421 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:20:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-10T21:20:14.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:20:14.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:14.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:15.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:20:15.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:15.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:15.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:20:15.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:15.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:15.315 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:20:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:20:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:20:15.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-10T21:20:15.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:20:15.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:20:15.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:20:16.026 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:20:16.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:20:16.553 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:20:16.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:20:16.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:20:16.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:20:16.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-10T21:20:16.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:20:16.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:20:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-10T21:20:16.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:20:16.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:20:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:20:16.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:20:16.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:20:16.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:20:17.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:20:17.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:17.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:20:17.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:20:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:20:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:20:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:20:17.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:19.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:20:19.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:20:19.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:20:19.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:20:19.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:20:19.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:20:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:20:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:20:19.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:20:19.893 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:19.890+0000 7f61d0c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:19.895 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:19.892+0000 7f61d0c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:19.897 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:19.894+0000 7f61d0c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:20.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:20.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:21.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:21.014+0000 7f61d0c33640 -1 Falling back to public interface 2024-09-10T21:20:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:20:21.425 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:21.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:21.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:21.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:22.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:22.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:22.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:20:22.719 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:20:22.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:22.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:22.870 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:22.865+0000 7f61d0c33640 -1 osd.1 36 log_to_monitors true 2024-09-10T21:20:23.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:24.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:24.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:24.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:20:24.081 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:20:24.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:24.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/3883734261,v1:127.0.0.1:6811/3883734261] [v2:127.0.0.1:6812/3883734261,v1:127.0.0.1:6813/3883734261] exists,up 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:20:24.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:20:24.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:20:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:20:24.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:20:24.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:20:24.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:20:24.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:20:24.691 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:20:24.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:20:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:20:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:20:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:24.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:20:25.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T21:20:25.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T21:20:25.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T21:20:25.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:25.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:20:25.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-10T21:20:25.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-10T21:20:25.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-176093659138' 2024-09-10T21:20:25.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:25.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:20:25.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-10T21:20:25.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-10T21:20:25.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-176093659138 2-64424509454' 2024-09-10T21:20:25.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:25.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T21:20:25.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:25.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:20:25.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T21:20:25.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:25.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T21:20:25.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T21:20:25.277 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836506 2024-09-10T21:20:25.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:25.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T21:20:25.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:25.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659138 2024-09-10T21:20:25.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:25.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:20:25.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659138 2024-09-10T21:20:25.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-10T21:20:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-09-10T21:20:25.578 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 176093659138 2024-09-10T21:20:25.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:25.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-10T21:20:25.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:25.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-10T21:20:25.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:25.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:20:25.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-10T21:20:25.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:25.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-10T21:20:25.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-10T21:20:25.884 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509454 2024-09-10T21:20:25.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:20:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-10T21:20:26.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:20:26.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:26.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:26.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:26.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:26.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:26.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:20:26.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:26.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:26.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:27.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:20:27.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:20:27.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:20:27.522 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:20:27.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:20:27.907 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:20:27.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:20:27.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:20:27.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:20:28.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-10T21:20:28.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:20:28.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:20:28.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1916e-05 s, 33.1 MB/s 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:20:28.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:20:28.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:20:28.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:20:28.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:20:30.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:20:30.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:20:30.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:20:30.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:20:30.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:20:30.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:20:30.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:20:30.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:20:31.005 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:31.001+0000 7f3ff8215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:31.008 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:31.003+0000 7f3ff8215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:31.009 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:31.005+0000 7f3ff8215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:31.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:20:31.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:20:31.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:20:31.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:20:31.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:20:31.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:31.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:20:31.263 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:20:31.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:31.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:20:31.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:31.880 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:31.876+0000 7f3ff8215640 -1 Falling back to public interface 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:32.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:20:32.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:33.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:33.614+0000 7f3ff8215640 -1 osd.0 44 log_to_monitors true 2024-09-10T21:20:33.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:33.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:33.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:20:33.859 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:20:33.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:33.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:20:34.145 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 49 up_thru 41 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/1659912144,v1:127.0.0.1:6803/1659912144] [v2:127.0.0.1:6804/1659912144,v1:127.0.0.1:6805/1659912144] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:20:34.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:20:34.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:20:34.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:20:34.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:20:34.147 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:20:34.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:20:34.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:20:34.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:20:34.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:34.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:20:34.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397506 2024-09-10T21:20:34.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397506 2024-09-10T21:20:34.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506' 2024-09-10T21:20:34.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:34.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:20:35.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-10T21:20:35.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-10T21:20:35.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-176093659141' 2024-09-10T21:20:35.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:35.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:20:35.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-10T21:20:35.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-10T21:20:35.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-176093659141 2-64424509457' 2024-09-10T21:20:35.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:35.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397506 2024-09-10T21:20:35.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:35.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:20:35.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397506 2024-09-10T21:20:35.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:35.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397506 2024-09-10T21:20:35.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-09-10T21:20:35.327 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 210453397506 2024-09-10T21:20:35.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:35.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 210453397506 2024-09-10T21:20:35.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:36.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:20:36.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:36.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397506 -lt 210453397506 2024-09-10T21:20:36.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:36.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659141 2024-09-10T21:20:36.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:36.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:20:36.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659141 2024-09-10T21:20:36.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:36.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-10T21:20:36.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-09-10T21:20:36.972 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 176093659141 2024-09-10T21:20:36.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:37.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659141 -lt 176093659141 2024-09-10T21:20:37.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:37.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-10T21:20:37.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:37.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:20:37.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-10T21:20:37.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:37.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-10T21:20:37.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-10T21:20:37.261 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509457 2024-09-10T21:20:37.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:20:37.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-10T21:20:37.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:20:37.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:37.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:37.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:20:37.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:37.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:37.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:37.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:37.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:37.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:37.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:38.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T21:20:38.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:38.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:38.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:38.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-10T21:20:38.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-10T21:20:38.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:20:38.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T21:20:38.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:20:38.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:38.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:38.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T21:20:38.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:38.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:38.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:39.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-10T21:20:39.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:20:39.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:20:39.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:20:39.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:20:39.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:20:39.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:20:39.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:20:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:20:39.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:39.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:39.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:39.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:39.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:39.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:39.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:40.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:20:40.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:40.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:40.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:40.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:20:40.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:20:40.826 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:20:40.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:20:41.362 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:20:41.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:20:41.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:20:41.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:20:41.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-10T21:20:41.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:20:41.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:20:41.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:20:41.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:20:41.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:20:41.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:20:41.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:20:41.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:20:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:20:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:20:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-10T21:20:43.876 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:20:44.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:20:44.394 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:20:44.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:20:44.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:20:44.396 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:20:44.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:20:44.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:20:44.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:20:44.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:20:44.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:20:44.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:20:44.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:44.421+0000 7fa09f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:44.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:44.424+0000 7fa09f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:44.428 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:44.425+0000 7fa09f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:20:44.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:44.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:44.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:45.559 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:45.556+0000 7fa09f319640 -1 Falling back to public interface 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:45.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:46.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:47.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:47.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:20:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:20:47.770+0000 7fa09f319640 -1 osd.1 53 log_to_monitors true 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:20:48.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:20:48.823 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [41,53) [v2:127.0.0.1:6810/2612868906,v1:127.0.0.1:6811/2612868906] [v2:127.0.0.1:6812/2612868906,v1:127.0.0.1:6813/2612868906] exists,up 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:20:48.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:20:48.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:20:48.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:20:48.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:20:48.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:20:48.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:20:48.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:20:48.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:20:48.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:20:48.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:20:48.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:20:48.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:20:48.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:20:48.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:49.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:20:49.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397510 2024-09-10T21:20:49.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397510 2024-09-10T21:20:49.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510' 2024-09-10T21:20:49.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:49.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:20:49.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103170 2024-09-10T21:20:49.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103170 2024-09-10T21:20:49.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170' 2024-09-10T21:20:49.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:20:49.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:20:49.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-10T21:20:49.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-10T21:20:49.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170 2-64424509461' 2024-09-10T21:20:49.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:49.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397510 2024-09-10T21:20:49.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:49.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:20:49.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397510 2024-09-10T21:20:49.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:49.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397510 2024-09-10T21:20:49.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397510' 2024-09-10T21:20:49.672 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 210453397510 2024-09-10T21:20:49.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:49.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397509 -lt 210453397510 2024-09-10T21:20:49.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:50.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:20:50.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:51.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397509 -lt 210453397510 2024-09-10T21:20:51.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:20:52.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:20:52.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:20:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397510 -lt 210453397510 2024-09-10T21:20:52.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:52.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103170 2024-09-10T21:20:52.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:52.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:20:52.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103170 2024-09-10T21:20:52.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103170 2024-09-10T21:20:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-09-10T21:20:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 249108103170 2024-09-10T21:20:52.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:20:52.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103170 -lt 249108103170 2024-09-10T21:20:52.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:20:52.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-10T21:20:52.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:20:52.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:20:52.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-10T21:20:52.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:20:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-10T21:20:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-10T21:20:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509461 2024-09-10T21:20:52.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:20:53.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-10T21:20:53.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:20:53.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:53.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:53.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:20:53.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:53.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:20:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:20:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:20:54.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:20:54.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:54.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:54.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:54.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:54.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:54.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:54.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:54.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:20:54.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:54.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:54.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:20:54.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:20:54.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:20:54.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:20:54.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:20:55.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:20:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:20:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:20:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:20:55.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:20:55.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:20:55.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:20:55.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:20:55.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:20:55.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:20:55.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:20:55.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:20:55.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:20:55.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:20:55.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:20:55.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:20:55.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:20:55.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:20:55.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:20:55.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:20:55.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:20:56.343 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:20:56.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:20:56.854 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:20:56.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:20:56.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:20:56.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:20:57.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-10T21:20:57.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:20:57.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:20:57.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-10T21:20:57.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:20:57.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:20:57.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:20:57.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:20:57.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:20:57.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:20:57.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:20:57.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:20:57.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:20:57.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:20:57.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-10T21:21:00.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:21:00.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:21:00.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:21:00.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:21:00.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:21:00.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:21:00.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:00.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:21:00.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:21:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:21:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:21:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:21:00.333 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:21:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:21:00.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:21:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:21:00.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:21:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:21:00.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:21:00.355 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:00.352+0000 7fc91e213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:00.358 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:00.355+0000 7fc91e213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:00.356+0000 7fc91e213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:21:00.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:21:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:21:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:00.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:00.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:01.707 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:01.704+0000 7fc91e213640 -1 Falling back to public interface 2024-09-10T21:21:01.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:01.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:01.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:21:01.887 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:01.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:01.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:02.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:03.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:03.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:03.300+0000 7fc91e213640 -1 osd.0 62 log_to_monitors true 2024-09-10T21:21:03.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:04.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:04.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:21:04.452 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:21:04.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:04.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/3920506549,v1:127.0.0.1:6803/3920506549] [v2:127.0.0.1:6804/3920506549,v1:127.0.0.1:6805/3920506549] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:21:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:21:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:21:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:21:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:21:04.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:21:05.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:21:05.272 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:05.272 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:21:05.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:21:05.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:05.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:21:05.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808834 2024-09-10T21:21:05.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808834 2024-09-10T21:21:05.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834' 2024-09-10T21:21:05.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:05.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:21:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103174 2024-09-10T21:21:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103174 2024-09-10T21:21:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174' 2024-09-10T21:21:05.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:05.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174 2-64424509465' 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808834 2024-09-10T21:21:05.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:05.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:21:05.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808834 2024-09-10T21:21:05.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:05.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808834 2024-09-10T21:21:05.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-09-10T21:21:05.583 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 287762808834 2024-09-10T21:21:05.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:05.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808834 2024-09-10T21:21:05.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:06.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:21:06.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:07.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808834 2024-09-10T21:21:07.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:08.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:21:08.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:08.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808834 -lt 287762808834 2024-09-10T21:21:08.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:08.421 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103174 2024-09-10T21:21:08.421 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:08.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:21:08.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103174 2024-09-10T21:21:08.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:08.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103174 2024-09-10T21:21:08.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103174' 2024-09-10T21:21:08.424 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 249108103174 2024-09-10T21:21:08.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:21:08.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103174 -lt 249108103174 2024-09-10T21:21:08.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:08.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-10T21:21:08.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:08.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:21:08.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-10T21:21:08.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:08.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-10T21:21:08.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-10T21:21:08.708 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 64424509465 2024-09-10T21:21:08.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:08.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-10T21:21:08.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:21:08.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:08.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:09.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:21:09.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:09.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:09.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:09.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:09.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:09.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:09.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:09.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:21:09.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:09.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:09.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:21:09.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:21:09.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:21:10.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:21:10.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:21:10.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:21:10.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:21:10.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:21:10.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:21:10.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:21:10.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:21:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:21:12.721 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:21:12.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:21:12.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:21:12.723 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:21:12.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:21:12.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:21:12.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:21:12.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:21:12.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:21:12.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:21:12.750 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:12.747+0000 7f15a2c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:12.753 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:12.750+0000 7f15a2c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:12.754 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:12.751+0000 7f15a2c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:13.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:13.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:13.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:14.115 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:14.112+0000 7f15a2c1e640 -1 Falling back to public interface 2024-09-10T21:21:14.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:14.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:14.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:21:14.287 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:14.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:14.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:14.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:15.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:15.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:15.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:21:15.570 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:21:15.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:15.839 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:15.836+0000 7f15a2c1e640 -1 osd.2 69 log_to_monitors true 2024-09-10T21:21:15.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:16.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:16.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:16.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:21:16.855 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:21:16.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:16.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1856036427,v1:127.0.0.1:6819/1856036427] [v2:127.0.0.1:6820/1856036427,v1:127.0.0.1:6821/1856036427] exists,up 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:21:17.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:21:17.307 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:21:17.307 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:21:17.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:21:17.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:21:17.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:21:17.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:21:17.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:21:17.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:21:17.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:21:17.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:21:17.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:17.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:21:17.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808837 2024-09-10T21:21:17.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808837 2024-09-10T21:21:17.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837' 2024-09-10T21:21:17.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:17.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:21:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103177 2024-09-10T21:21:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103177 2024-09-10T21:21:17.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177' 2024-09-10T21:21:17.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:17.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:21:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-10T21:21:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-10T21:21:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177 2-317827579906' 2024-09-10T21:21:18.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:18.086 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808837 2024-09-10T21:21:18.086 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:18.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:21:18.088 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808837 2024-09-10T21:21:18.088 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808837 2024-09-10T21:21:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-09-10T21:21:18.089 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 287762808837 2024-09-10T21:21:18.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:18.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808836 -lt 287762808837 2024-09-10T21:21:18.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:19.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:21:19.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:19.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808838 -lt 287762808837 2024-09-10T21:21:19.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:19.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103177 2024-09-10T21:21:19.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:19.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:21:19.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103177 2024-09-10T21:21:19.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:19.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103177 2024-09-10T21:21:19.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-09-10T21:21:19.651 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 249108103177 2024-09-10T21:21:19.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:21:19.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103178 -lt 249108103177 2024-09-10T21:21:19.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:19.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-317827579906 2024-09-10T21:21:19.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:19.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:21:19.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-317827579906 2024-09-10T21:21:19.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:19.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-10T21:21:19.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-09-10T21:21:19.931 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 317827579906 2024-09-10T21:21:19.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:20.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-10T21:21:20.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:21:20.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:20.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:20.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:20.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:20.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:20.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:21:20.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:20.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:20.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:21:21.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:21:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:21:21.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:21:21.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:21:23.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:21:23.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:21:23.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:21:23.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:21:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:21:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:21:23.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:21:23.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:21:23.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:21:23.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:21:23.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:21:23.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:21:23.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:23.653+0000 7f4a3800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:23.660 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:23.656+0000 7f4a3800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:23.661 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:23.657+0000 7f4a3800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:21:23.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:23.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:24.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:24.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:24.779+0000 7f4a3800b640 -1 Falling back to public interface 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:25.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:25.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:26.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:26.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:26.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:21:26.473 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:21:26.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:26.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:26.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:26.559+0000 7f4a3800b640 -1 osd.2 77 log_to_monitors true 2024-09-10T21:21:26.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:27.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/1097790741,v1:127.0.0.1:6819/1097790741] [v2:127.0.0.1:6820/1097790741,v1:127.0.0.1:6821/1097790741] exists,up 58fa2c32-d12b-4e32-9ea5-6cde9f9f5b38 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:21:28.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:21:28.151 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:21:28.151 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:21:28.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:21:28.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:21:28.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:21:28.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:21:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:21:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:21:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:21:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:21:28.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:21:28.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:21:28.518 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:28.518 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:21:28.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:21:28.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:28.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:21:28.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808841 2024-09-10T21:21:28.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808841 2024-09-10T21:21:28.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841' 2024-09-10T21:21:28.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:28.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:21:28.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103181 2024-09-10T21:21:28.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103181 2024-09-10T21:21:28.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181' 2024-09-10T21:21:28.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:28.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:21:28.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-10T21:21:28.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-10T21:21:28.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181 2-352187318274' 2024-09-10T21:21:28.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:28.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808841 2024-09-10T21:21:28.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:28.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:21:28.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808841 2024-09-10T21:21:28.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:28.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808841 2024-09-10T21:21:28.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-09-10T21:21:28.832 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 287762808841 2024-09-10T21:21:28.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:29.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808839 -lt 287762808841 2024-09-10T21:21:29.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:30.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:21:30.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:30.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808841 -lt 287762808841 2024-09-10T21:21:30.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:30.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103181 2024-09-10T21:21:30.391 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:30.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:21:30.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103181 2024-09-10T21:21:30.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:30.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103181 2024-09-10T21:21:30.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103181' 2024-09-10T21:21:30.393 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 249108103181 2024-09-10T21:21:30.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:21:30.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103181 -lt 249108103181 2024-09-10T21:21:30.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:30.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-10T21:21:30.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:30.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:21:30.670 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-10T21:21:30.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:30.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-10T21:21:30.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-10T21:21:30.672 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318274 2024-09-10T21:21:30.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:30.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-10T21:21:30.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:21:30.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:30.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:31.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:31.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:21:31.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:31.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:31.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:31.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:21:31.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:21:31.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:21:31.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-10T21:21:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:21:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-10T21:21:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-10T21:21:31.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:21:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:21:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-10T21:21:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:21:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:21:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:21:32.309 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:21:32.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:21:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:21:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:21:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:21:32.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:21:33.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-10T21:21:33.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:21:33.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:21:33.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-10T21:21:33.020 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records in 2024-09-10T21:21:33.020 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records out 2024-09-10T21:21:33.020 INFO:tasks.workunit.client.0.smithi181.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.3882e-05 s, 55.4 MB/s 2024-09-10T21:21:33.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:21:33.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:21:33.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:21:33.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:21:33.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:21:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:21:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:21:35.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:21:35.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:21:35.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:21:35.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:21:35.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:21:35.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:21:35.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:21:35.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:21:35.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:35.663+0000 7f46a3c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:35.668 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:35.665+0000 7f46a3c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:35.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:35.666+0000 7f46a3c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:21:35.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:35.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:21:36.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:36.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:36.775+0000 7f46a3c18640 -1 Falling back to public interface 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:37.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:21:37.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:38.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:21:38.622 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:38.619+0000 7f46a3c18640 -1 osd.1 86 log_to_monitors true 2024-09-10T21:21:38.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:39.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:39.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:39.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:21:39.770 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:21:39.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:39.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [58,86) [v2:127.0.0.1:6810/819855696,v1:127.0.0.1:6811/819855696] [v2:127.0.0.1:6812/819855696,v1:127.0.0.1:6813/819855696] exists,up 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:21:40.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:21:40.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:21:40.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:21:40.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:21:40.581 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:40.581 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:21:40.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:21:40.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:40.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:21:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808844 2024-09-10T21:21:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808844 2024-09-10T21:21:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808844' 2024-09-10T21:21:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:40.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:21:40.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-10T21:21:40.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-10T21:21:40.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808844 1-390842023938' 2024-09-10T21:21:40.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:40.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:21:41.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318277 2024-09-10T21:21:41.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318277 2024-09-10T21:21:41.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808844 1-390842023938 2-352187318277' 2024-09-10T21:21:41.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:41.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808844 2024-09-10T21:21:41.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:41.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:21:41.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808844 2024-09-10T21:21:41.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:41.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808844 2024-09-10T21:21:41.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808844' 2024-09-10T21:21:41.012 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 287762808844 2024-09-10T21:21:41.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:41.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808843 -lt 287762808844 2024-09-10T21:21:41.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:42.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:21:42.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:42.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808844 -lt 287762808844 2024-09-10T21:21:42.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:42.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023938 2024-09-10T21:21:42.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:42.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:21:42.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023938 2024-09-10T21:21:42.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:42.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-10T21:21:42.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-09-10T21:21:42.573 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 390842023938 2024-09-10T21:21:42.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:21:42.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-10T21:21:42.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:42.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318277 2024-09-10T21:21:42.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:42.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:21:42.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318277 2024-09-10T21:21:42.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:42.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318277 2024-09-10T21:21:42.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318277' 2024-09-10T21:21:42.852 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318277 2024-09-10T21:21:42.853 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:43.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318277 -lt 352187318277 2024-09-10T21:21:43.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:21:43.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:43.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:43.471 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:43.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:21:43.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:43.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:43.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-10T21:21:44.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:21:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:21:44.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:21:44.463 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:21:44.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:21:44.973 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:21:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:21:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:21:44.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:21:45.219 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-10T21:21:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-10T21:21:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:21:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:21:45.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:21:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:21:45.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:21:47.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:21:47.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:21:47.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:21:47.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:21:47.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:21:47.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:21:47.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:21:47.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:47.800+0000 7f0c29aed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:47.802+0000 7f0c29aed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:47.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:47.803+0000 7f0c29aed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:21:48.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:48.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:48.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:49.158 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:49.155+0000 7f0c29aed640 -1 Falling back to public interface 2024-09-10T21:21:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:49.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:49.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:21:49.340 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:49.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:49.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:49.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:50.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:50.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:50.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:21:50.629 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:21:50.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:50.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:50.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:21:51.035 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:21:51.032+0000 7f0c29aed640 -1 osd.0 95 log_to_monitors true 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:21:51.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:21:52.335 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 100 up_thru 91 down_at 96 last_clean_interval [67,95) [v2:127.0.0.1:6802/1402413523,v1:127.0.0.1:6803/1402413523] [v2:127.0.0.1:6804/1402413523,v1:127.0.0.1:6805/1402413523] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:21:52.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:21:52.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:21:52.336 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:21:52.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:21:52.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:21:52.882 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:21:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:21:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:21:52.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:52.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:21:53.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-10T21:21:53.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-10T21:21:53.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602' 2024-09-10T21:21:53.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:53.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:21:53.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023941 2024-09-10T21:21:53.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023941 2024-09-10T21:21:53.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023941' 2024-09-10T21:21:53.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:21:53.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:21:53.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318281 2024-09-10T21:21:53.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318281 2024-09-10T21:21:53.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023941 2-352187318281' 2024-09-10T21:21:53.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:53.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729602 2024-09-10T21:21:53.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:53.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:21:53.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729602 2024-09-10T21:21:53.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:53.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-10T21:21:53.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-09-10T21:21:53.369 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 429496729602 2024-09-10T21:21:53.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:21:53.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729602 -lt 429496729602 2024-09-10T21:21:53.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:53.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023941 2024-09-10T21:21:53.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:53.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:21:53.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023941 2024-09-10T21:21:53.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:53.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023941 2024-09-10T21:21:53.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023941' 2024-09-10T21:21:53.669 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 390842023941 2024-09-10T21:21:53.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:21:53.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023941 -lt 390842023941 2024-09-10T21:21:53.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:21:53.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318281 2024-09-10T21:21:53.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:21:53.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:21:53.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318281 2024-09-10T21:21:53.946 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:21:53.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318281 2024-09-10T21:21:53.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318281' 2024-09-10T21:21:53.948 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318281 2024-09-10T21:21:53.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318280 -lt 352187318281 2024-09-10T21:21:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:21:55.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:21:55.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:21:55.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318281 -lt 352187318281 2024-09-10T21:21:55.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:21:55.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:55.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:21:55.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:55.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:55.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:55.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:55.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:55.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:55.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:21:56.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:56.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:56.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:56.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:21:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-10T21:21:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:21:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-10T21:21:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:21:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:21:56.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:21:56.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:56.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:56.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:56.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:56.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:56.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:56.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:56.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:21:56.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:56.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:56.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:21:57.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:21:57.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:21:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:21:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:21:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:21:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:21:57.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:21:57.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:21:57.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:21:57.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:21:58.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:21:58.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:21:58.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:21:58.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:21:58.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:21:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:21:58.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:22:00.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:00.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:22:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:22:00.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:22:00.956 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:22:00.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:22:00.984 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:00.981+0000 7fac9900f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:00.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:00.984+0000 7fac9900f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:00.987 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:00.985+0000 7fac9900f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:01.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:01.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:22:01.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:02.351 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:02.348+0000 7fac9900f640 -1 Falling back to public interface 2024-09-10T21:22:02.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:02.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:02.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:22:02.517 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:02.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:02.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:22:02.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:03.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:03.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:03.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:22:03.803 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:22:03.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:03.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:22:04.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:04.188 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:04.185+0000 7fac9900f640 -1 osd.1 102 log_to_monitors true 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:05.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/2862105973,v1:127.0.0.1:6811/2862105973] [v2:127.0.0.1:6812/2862105973,v1:127.0.0.1:6813/2862105973] exists,up 9e7e8ae8-085f-4955-bdbd-cfb1c64025b8 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:22:05.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:22:05.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:22:05.416 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:22:05.416 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:22:05.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:22:05.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:22:05.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:22:05.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:22:05.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:22:05.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:22:05.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:22:05.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:22:05.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:05.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:22:05.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729605 2024-09-10T21:22:05.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729605 2024-09-10T21:22:05.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605' 2024-09-10T21:22:05.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:05.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:22:06.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-10T21:22:06.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-10T21:22:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-09-10T21:22:06.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:06.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:22:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318284 2024-09-10T21:22:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318284 2024-09-10T21:22:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-352187318284' 2024-09-10T21:22:06.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:06.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729605 2024-09-10T21:22:06.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:06.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:22:06.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729605 2024-09-10T21:22:06.148 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:06.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729605 2024-09-10T21:22:06.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-09-10T21:22:06.149 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 429496729605 2024-09-10T21:22:06.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729604 -lt 429496729605 2024-09-10T21:22:06.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:22:07.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:22:07.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:07.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729606 -lt 429496729605 2024-09-10T21:22:07.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:07.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500674 2024-09-10T21:22:07.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:07.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:22:07.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500674 2024-09-10T21:22:07.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:07.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-10T21:22:07.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-09-10T21:22:07.728 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500674 2024-09-10T21:22:07.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:22:08.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-10T21:22:08.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:08.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318284 2024-09-10T21:22:08.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:08.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:22:08.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318284 2024-09-10T21:22:08.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:08.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318284 2024-09-10T21:22:08.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318284' 2024-09-10T21:22:08.010 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318284 2024-09-10T21:22:08.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:22:08.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318285 -lt 352187318284 2024-09-10T21:22:08.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:22:08.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:08.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:22:08.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:22:08.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:22:08.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:22:08.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:22:08.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:22:08.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:08.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:09.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:22:09.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:22:09.753 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:22:09.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:22:10.154 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:22:10.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:22:10.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:22:10.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:22:10.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-10T21:22:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-10T21:22:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-10T21:22:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-10T21:22:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-10T21:22:10.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:22:10.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:22:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:22:10.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:22:10.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:22:12.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:12.815 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:22:12.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:22:12.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:22:12.817 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:22:12.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:22:12.846 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:12.842+0000 7ffa76a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:12.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:12.844+0000 7ffa76a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:12.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:12.846+0000 7ffa76a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:13.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:13.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:13.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:13.980+0000 7ffa76a2a640 -1 Falling back to public interface 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:14.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:14.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:15.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:15.661+0000 7ffa76a2a640 -1 osd.0 111 log_to_monitors true 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:15.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:15.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:22:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:22:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:16.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:17.371 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/549248512,v1:127.0.0.1:6803/549248512] [v2:127.0.0.1:6804/549248512,v1:127.0.0.1:6805/549248512] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:22:17.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:22:17.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:22:17.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:22:17.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:22:17.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:22:17.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:22:17.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:22:17.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:22:17.373 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:22:17.373 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:22:17.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:22:17.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:22:17.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:22:17.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:22:17.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:22:17.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:22:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:22:17.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:22:17.465 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:22:17.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:22:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:22:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:22:17.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:17.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:22:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=498216206338 2024-09-10T21:22:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 498216206338 2024-09-10T21:22:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338' 2024-09-10T21:22:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:17.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:22:18.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500677 2024-09-10T21:22:18.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500677 2024-09-10T21:22:18.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677' 2024-09-10T21:22:18.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:18.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:22:18.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318288 2024-09-10T21:22:18.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318288 2024-09-10T21:22:18.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677 2-352187318288' 2024-09-10T21:22:18.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:18.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-498216206338 2024-09-10T21:22:18.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:18.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:22:18.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-498216206338 2024-09-10T21:22:18.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:18.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=498216206338 2024-09-10T21:22:18.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-09-10T21:22:18.132 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 498216206338 2024-09-10T21:22:18.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:18.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 498216206338 2024-09-10T21:22:18.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:22:19.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:22:19.406 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:19.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-10T21:22:19.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:19.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500677 2024-09-10T21:22:19.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:19.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:22:19.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500677 2024-09-10T21:22:19.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:19.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500677 2024-09-10T21:22:19.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500677' 2024-09-10T21:22:19.736 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500677 2024-09-10T21:22:19.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:22:20.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500677 -lt 459561500677 2024-09-10T21:22:20.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:20.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318288 2024-09-10T21:22:20.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:20.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:22:20.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318288 2024-09-10T21:22:20.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:20.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318288 2024-09-10T21:22:20.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318288' 2024-09-10T21:22:20.017 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318288 2024-09-10T21:22:20.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:22:20.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318288 -lt 352187318288 2024-09-10T21:22:20.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:22:20.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:20.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:20.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:22:20.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:22:20.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:22:20.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:22:20.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:20.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:22:21.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:22:21.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:22:21.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:22:21.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:22:21.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:22:21.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:22:21.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:22:21.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:22:21.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:22:21.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-10T21:22:24.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:22:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:24.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:22:24.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:22:24.116 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:22:24.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:22:24.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:22:24.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:22:24.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:22:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:24.143+0000 7f8ddfceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:24.147 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:24.145+0000 7f8ddfceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:24.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:24.146+0000 7f8ddfceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:24.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:22:24.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:22:24.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:22:24.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:22:24.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:22:24.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:24.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:22:24.404 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:22:24.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:24.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:24.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:25.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:25.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:25.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:22:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:25.757 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:25.755+0000 7f8ddfceb640 -1 Falling back to public interface 2024-09-10T21:22:25.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:26.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:27.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:27.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:27.801+0000 7f8ddfceb640 -1 osd.0 118 log_to_monitors true 2024-09-10T21:22:28.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:28.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:28.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:22:28.264 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:22:28.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:28.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/91910223,v1:127.0.0.1:6803/91910223] [v2:127.0.0.1:6804/91910223,v1:127.0.0.1:6805/91910223] exists,up 8d63ab54-734d-49f1-bdde-c448701618bc 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:22:28.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:22:28.716 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:22:28.716 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:22:28.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:22:28.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:22:28.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:22:28.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:22:29.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:22:29.162 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:22:29.162 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:22:29.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:22:29.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:29.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:22:29.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-10T21:22:29.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-10T21:22:29.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410' 2024-09-10T21:22:29.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:29.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:22:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500681 2024-09-10T21:22:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500681 2024-09-10T21:22:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681' 2024-09-10T21:22:29.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:22:29.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:22:29.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318291 2024-09-10T21:22:29.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318291 2024-09-10T21:22:29.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681 2-352187318291' 2024-09-10T21:22:29.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:29.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-528280977410 2024-09-10T21:22:29.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:29.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:22:29.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-528280977410 2024-09-10T21:22:29.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:29.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-10T21:22:29.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-09-10T21:22:29.642 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 528280977410 2024-09-10T21:22:29.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:30.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 528280977410 2024-09-10T21:22:30.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:22:31.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:22:31.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 528280977410 2024-09-10T21:22:31.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:22:32.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:22:32.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:22:32.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-10T21:22:32.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:32.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500681 2024-09-10T21:22:32.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:32.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:22:32.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500681 2024-09-10T21:22:32.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:32.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500681 2024-09-10T21:22:32.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500681' 2024-09-10T21:22:32.586 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 459561500681 2024-09-10T21:22:32.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:22:32.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500681 -lt 459561500681 2024-09-10T21:22:32.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:22:32.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318291 2024-09-10T21:22:32.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:22:32.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:22:32.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318291 2024-09-10T21:22:32.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:22:32.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318291 2024-09-10T21:22:32.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318291' 2024-09-10T21:22:32.867 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.2 seq 352187318291 2024-09-10T21:22:32.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:22:33.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318291 -lt 352187318291 2024-09-10T21:22:33.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:22:33.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:33.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:22:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:22:33.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:22:33.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:22:33.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:22:33.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:22:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:22:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:22:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:22:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-10T21:22:34.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-10T21:22:34.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:22:34.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-10T21:22:34.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-10T21:22:34.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:34.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:34.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:34.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-10T21:22:34.946 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-10T21:22:34.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:34.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:22:34.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:34.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:22:34.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:34.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:35.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:35.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:36.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:36.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:36.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:36.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:36.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:36.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:37.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:37.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:37.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:37.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:37.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:37.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:37.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:37.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:37.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:39.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:39.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:40.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:40.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:40.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:40.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:40.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:40.085 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:40.086 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:40.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:40.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:41.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:22:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:22:41.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:41.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:22:41.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:22:35.176663+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:41.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:22:41.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-10T21:22:41.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:22:41.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-10T21:22:41.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:22:41.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-10T21:22:41.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-10T21:22:41.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:22:41.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:22:41.718 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:22:41.719 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:22:41.719 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:22:41.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:22:41.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:22:41.741 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:22:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:22:41.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:22:41.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:22:41.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:22:41.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:41.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:22:42.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:42.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-10T21:22:42.294 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-10T21:22:42.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-10T21:20:05.125273+0000 last_deep_scrub_stamp 2024-09-10T21:22:42.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:22:42.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:42.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-10T21:22:42.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:42.307 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:22:42.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:42.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:43.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:43.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:22:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:22:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:43.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:22:43.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:43.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:44.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:44.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:22:44.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:44.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:22:44.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:44.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:22:45.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:20:05.125273+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:45.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:22:46.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:22:46.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:22:46.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:22:46.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:22:46.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:22:46.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:22:46.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:22:46.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:22:43.355097+0000 '>' 2024-09-10T21:20:05.125273+0000 2024-09-10T21:22:46.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:22:46.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-10T21:22:46.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:22:46.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-10T21:22:46.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:22:46.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-10T21:22:46.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-10T21:22:46.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:22:46.494 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:22:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:22:46.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:22:46.518 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:22:46.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-10T21:22:46.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:22:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:22:46.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:22:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-10T21:22:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-10T21:22:46.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-10T21:22:46.934 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' removed 2024-09-10T21:22:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:22:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:22:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:22:46.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:22:46.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:22:46.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:22:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:22:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:22:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:22:47.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:22:47.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:22:47.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:22:47.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:22:47.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:22:47.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:22:47.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:22:47.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:22:47.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:22:47.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:22:47.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:22:47.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:22:47.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:22:47.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:22:47.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:22:47.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:22:47.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:22:47.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:22:47.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:22:47.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:22:47.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:22:47.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:22:47.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:22:47.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:22:47.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:22:47.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:22:47.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:22:47.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:22:47.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:22:47.101 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:22:47.101 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.101 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:22:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:22:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:22:47.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:22:47.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:22:47.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.104 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-10T21:22:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:22:47.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-10T21:22:47.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:22:47.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:47.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:22:47.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:22:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:22:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:22:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:22:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:22:47.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:22:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:22:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:22:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:22:47.745 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:22:47.746 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.746 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.746 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:22:47.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:22:47.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:22:47.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:47.825 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:22:47.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:22:47.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:22:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:22:48.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:48.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:22:48.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:22:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:22:48.752 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:22:48.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:22:48.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:22:48.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:22:48.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:22:48.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 788989a4-ba89-4ed0-84aa-663c6f11bae5' 2024-09-10T21:22:48.755 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:22:48.755 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:22:48.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCouOBmQbHQLRAAP00Kly2iG6M771dOaq07Ow== 2024-09-10T21:22:48.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCouOBmQbHQLRAAP00Kly2iG6M771dOaq07Ow=="}' 2024-09-10T21:22:48.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 788989a4-ba89-4ed0-84aa-663c6f11bae5 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:22:48.976 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:22:48.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:22:48.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCouOBmQbHQLRAAP00Kly2iG6M771dOaq07Ow== --osd-uuid 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:22:49.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:49.015+0000 7f1ba2413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:49.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:49.024+0000 7f1ba2413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:49.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:49.026+0000 7f1ba2413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:49.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:49.026+0000 7f1ba2413640 -1 bdev(0x563d146bbc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:22:49.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:49.026+0000 7f1ba2413640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:22:54.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:22:54.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:22:54.641 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:22:54.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:22:54.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:22:55.002 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:22:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:22:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:22:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:22:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:22:55.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:22:55.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:55.027+0000 7f809fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:55.031 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:55.028+0000 7f809fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:55.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:55.030+0000 7f809fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:22:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:22:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:22:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:22:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:55.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:56.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:56.399+0000 7f809fb45640 -1 Falling back to public interface 2024-09-10T21:22:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:56.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:56.557 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:22:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:22:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:56.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:57.716 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:22:57.713+0000 7f809fb45640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:22:57.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:57.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:57.847 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:22:57.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:22:57.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:57.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:58.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:22:59.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:22:59.168 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:22:59.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:22:59.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:22:59.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:22:59.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:22:59.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:00.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:00.500 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:23:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:23:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:00.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:00.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:01.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:01.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:01.791 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:23:01.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:23:01.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:01.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:02.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:03.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:03.080 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:23:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:23:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:03.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:03.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:04.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:04.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:04.448 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:23:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:23:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:04.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:04.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:05.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:05.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:05.733 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:23:05.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:23:05.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:05.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:06.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:07.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:07.022 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:23:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:23:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:07.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:07.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:08.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:08.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:08.315 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:23:08.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:23:08.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:08.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:08.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:09.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:09.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:09.695 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:23:09.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:23:09.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:09.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:09.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:10.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:10.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:23:10.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:23:10.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:10.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:11.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:12.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:12.273 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:23:12.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:12.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:23:12.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:12.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:12.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:13.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:13.647 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:23:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:23:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:13.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:14.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:15.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:15.236 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:23:15.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:23:15.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:15.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:16.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:17.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:17.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:17.031 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:23:17.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:23:17.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:17.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:17.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:18.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:18.947 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:23:18.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:18.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:23:18.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:18.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:19.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:20.349 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:23:20.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:23:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:20.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:23:20.635 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3724250141,v1:127.0.0.1:6803/3724250141] [v2:127.0.0.1:6804/3724250141,v1:127.0.0.1:6805/3724250141] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:23:20.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:23:20.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:23:20.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:23:20.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:23:20.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6765db78-c000-4cda-9031-a1dd5c80194a' 2024-09-10T21:23:20.642 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:23:20.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:23:20.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIuOBmfc8OJxAAv3Z14rsLFuVvPm4Zka/ouA== 2024-09-10T21:23:20.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIuOBmfc8OJxAAv3Z14rsLFuVvPm4Zka/ouA=="}' 2024-09-10T21:23:20.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6765db78-c000-4cda-9031-a1dd5c80194a -i td/osd-scrub-repair/1/new.json 2024-09-10T21:23:21.059 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:23:21.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:23:21.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIuOBmfc8OJxAAv3Z14rsLFuVvPm4Zka/ouA== --osd-uuid 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:23:21.100 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:21.098+0000 7feec7e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:21.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:21.132+0000 7feec7e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:21.135 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:21.133+0000 7feec7e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:21.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:21.133+0000 7feec7e31640 -1 bdev(0x55bc61df3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:23:21.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:21.133+0000 7feec7e31640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:23:26.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:23:26.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:23:26.835 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:23:26.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:23:26.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:23:27.196 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:23:27.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:23:27.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:23:27.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:23:27.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:23:27.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:23:27.224 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:27.221+0000 7f6e33616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:27.226 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:27.223+0000 7f6e33616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:27.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:27.224+0000 7f6e33616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:23:27.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:27.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:23:27.483 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:23:27.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:27.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:27.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:28.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:28.757 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:23:28.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:28.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:23:28.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:28.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:28.845 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:28.843+0000 7f6e33616640 -1 Falling back to public interface 2024-09-10T21:23:29.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:30.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:30.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:30.073 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:23:30.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:23:30.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:30.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:30.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:30.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:23:30.762+0000 7f6e33616640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:23:31.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:31.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:31.363 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:23:31.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:23:31.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:31.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:31.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:32.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:32.654 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:23:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:23:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:32.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:32.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:33.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:33.947 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:23:33.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:33.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:23:33.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:33.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:34.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:35.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:35.317 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:23:35.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:35.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:23:35.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:35.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:35.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:36.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:36.623 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:23:36.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:36.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:23:36.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:36.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:36.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:37.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:37.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:37.939 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:23:37.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:23:37.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:37.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:38.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:39.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:39.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:39.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:23:39.291 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:23:39.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:39.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:39.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:40.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:40.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:40.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:23:40.652 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:23:40.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:40.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:40.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:41.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:41.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:41.939 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:23:41.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:23:41.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:41.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:42.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:43.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:43.346 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:23:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:23:43.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:43.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:43.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:44.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:44.946 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:23:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:23:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:45.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:23:46.879 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:23:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:48.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:48.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:48.273 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:23:48.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:23:48.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:48.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:48.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:49.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:49.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:49.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:23:49.959 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:23:49.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:49.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:50.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:51.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:51.554 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:23:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:23:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:51.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:52.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:53.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:53.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:53.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:23:53.262 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:23:53.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:53.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:53.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:23:54.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:23:54.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:23:54.786 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:23:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:23:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:23:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:23:55.085 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1498962709,v1:127.0.0.1:6811/1498962709] [v2:127.0.0.1:6812/1498962709,v1:127.0.0.1:6813/1498962709] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:23:55.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:23:55.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:23:55.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:23:55.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-09-10T21:23:55.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:23:55.351 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:23:55.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:23:55.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:23:55.759 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:23:55.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:23:56.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:23:58.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-10T21:23:58.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:23:58.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:23:58.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:23:58.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:23:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:23:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:23:58.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:23:58.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:23:58.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:23:58.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:23:58.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:23:58.474 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:23:58.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:23:58.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:23:58.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:23:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-10T21:23:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-10T21:23:58.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-10T21:23:58.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:23:58.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:23:58.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-10T21:23:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-10T21:23:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-09-10T21:23:58.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:23:58.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-10T21:23:58.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:23:58.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:23:58.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-10T21:23:58.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:23:58.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-10T21:23:58.684 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836495 2024-09-10T21:23:58.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-10T21:23:58.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:23:58.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-10T21:23:58.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:23:58.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-10T21:23:58.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:23:58.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:23:58.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-10T21:23:58.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:23:58.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-10T21:23:58.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-10T21:23:58.988 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672965 2024-09-10T21:23:58.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:23:59.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-10T21:23:59.269 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:23:59.269 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:23:59.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:23:59.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:23:59.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:23:59.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:23:59.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:23:59.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:23:59.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:24:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:24:00.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:24:00.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:24:00.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-09-10T21:24:00.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-10T21:24:00.637 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' created 2024-09-10T21:24:00.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:24:01.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-09-10T21:24:01.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-10T21:24:02.277 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'csr_pool' created 2024-09-10T21:24:02.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:24:03.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-10T21:24:03.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:24:03.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:24:03.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:24:03.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:24:03.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:24:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:24:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:24:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:24:03.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:24:03.393 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:24:03.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:24:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:24:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:24:03.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:24:03.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:24:03.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-10T21:24:03.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-10T21:24:03.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-10T21:24:03.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:24:03.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:24:03.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-10T21:24:03.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-10T21:24:03.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-42949672967' 2024-09-10T21:24:03.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:24:03.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-10T21:24:03.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:24:03.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:24:03.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-10T21:24:03.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:24:03.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-10T21:24:03.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-10T21:24:03.898 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836497 2024-09-10T21:24:03.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:24:04.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836497 2024-09-10T21:24:04.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:24:05.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:24:05.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:24:05.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-10T21:24:05.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:24:05.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-10T21:24:05.463 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:24:05.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:24:05.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-10T21:24:05.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:24:05.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-10T21:24:05.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-10T21:24:05.465 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672967 2024-09-10T21:24:05.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:24:05.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672968 -lt 42949672967 2024-09-10T21:24:05.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:24:05.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:24:05.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:24:06.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:24:06.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:24:06.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:24:06.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:24:06.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:24:06.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:24:06.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:24:06.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:24:06.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:24:06.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:06.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-10T21:24:06.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:06.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:06.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:07.089 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:07.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:07.525 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:07.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:07.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:07.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:07.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-10T21:24:07.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-10T21:24:08.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:08.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-10T21:24:08.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-10T21:24:08.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-10T21:24:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:08.428 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:08.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:08.955 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:08.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:08.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:08.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:09.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-10T21:24:09.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-10T21:24:09.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:09.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:09.959 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:10.369 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:10.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:10.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:10.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:10.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-09-10T21:24:10.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-09-10T21:24:10.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:10.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:10.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:11.305 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:11.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:11.694 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:11.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:11.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:11.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:11.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-09-10T21:24:12.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-09-10T21:24:12.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:12.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-10T21:24:12.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-09-10T21:24:12.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:12.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:12.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-09-10T21:24:12.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:12.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:12.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:12.598 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:12.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:13.033 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:13.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-09-10T21:24:13.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-09-10T21:24:13.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:13.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:14.101 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:14.512 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:14.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:14.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:14.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:14.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-09-10T21:24:14.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-09-10T21:24:15.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:15.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:15.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:15.500 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:15.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:15.935 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:15.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:15.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:15.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:16.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-09-10T21:24:16.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-09-10T21:24:16.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:16.852 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:16.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:17.372 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:17.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:17.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-09-10T21:24:17.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-09-10T21:24:17.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:17.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:17.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:18.359 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:18.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:18.886 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:18.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:18.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:19.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-09-10T21:24:19.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-09-10T21:24:19.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:19.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-10T21:24:19.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:19.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:19.790 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:19.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:20.176 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:20.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:20.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:20.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:20.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-09-10T21:24:20.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-09-10T21:24:20.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:20.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:20.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:21.155 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:21.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:21.548 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:21.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:21.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-09-10T21:24:21.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:22.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-09-10T21:24:22.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:22.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:22.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:22.427 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:22.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:22.947 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:23.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-09-10T21:24:23.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:23.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:23.842 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:23.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:24.386 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:24.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-09-10T21:24:24.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:24.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:25.306 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:25.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:25.842 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:25.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:25.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:25.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:26.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-09-10T21:24:26.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:26.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-09-10T21:24:26.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:26.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:26.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:26.745 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:26.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:27.289 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:27.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:27.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-09-10T21:24:27.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:27.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-09-10T21:24:27.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:27.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:27.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:28.160 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:28.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:28.571 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:28.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:28.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:28.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:28.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-09-10T21:24:28.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-09-10T21:24:29.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:29.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-10T21:24:29.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-09-10T21:24:29.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-09-10T21:24:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:29.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:29.542 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:29.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:30.052 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:30.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:30.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:30.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:30.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-09-10T21:24:30.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-09-10T21:24:30.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:30.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:30.880 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:30.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:31.316 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:31.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:31.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:31.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:31.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-09-10T21:24:31.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-09-10T21:24:31.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:31.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-10T21:24:31.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-09-10T21:24:31.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:24:31.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:24:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-09-10T21:24:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:24:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:24:31.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:24:32.195 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:24:32.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:24:32.572 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:24:32.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:24:32.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:24:32.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:24:32.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-09-10T21:24:33.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-09-10T21:24:33.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-09-10T21:24:33.222 INFO:tasks.workunit.client.0.smithi181.stderr:1025+0 records in 2024-09-10T21:24:33.222 INFO:tasks.workunit.client.0.smithi181.stderr:1025+0 records out 2024-09-10T21:24:33.222 INFO:tasks.workunit.client.0.smithi181.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.0018432 s, 569 MB/s 2024-09-10T21:24:33.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-09-10T21:24:33.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-09-10T21:24:33.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-10T21:24:33.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-10T21:24:33.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-10T21:24:33.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-10T21:24:33.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:24:33.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-10T21:24:33.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-09-10T21:24:33.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-10T21:24:33.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-10T21:24:33.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-10T21:24:33.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:24:33.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-09-10T21:24:33.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-09-10T21:24:33.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:24:33.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:24:33.960 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:24:33.961 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:24:33.961 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:24:33.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:24:33.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-10T21:24:33.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.52449/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-10T21:24:34.038 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:24:34.038 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-10T21:24:34.038 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:24:34.046 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:24:34.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:24:34.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-10T21:24:34.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-10T21:24:34.124 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:24:34.124 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-10T21:24:34.124 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:24:34.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-10T21:24:34.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-10T21:24:34.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:34.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:34.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:34.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:34.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:34.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:34.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-10T21:24:34.664 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-10T21:24:34.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-10T21:24:02.157944+0000 last_deep_scrub_stamp 2024-09-10T21:24:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:24:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-10T21:24:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:24:34.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:34.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:34.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:34.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:34.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:34.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:34.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:02.157944+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:34.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:24:35.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:24:35.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:35.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:35.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:35.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:35.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:35.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:36.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:02.157944+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:36.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:24:37.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:24:37.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:37.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:37.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:37.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:37.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:37.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:37.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:02.157944+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:37.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:24:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:24:38.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:38.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:38.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:38.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:38.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:38.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:38.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:02.157944+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:38.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:24:39.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:24:39.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:39.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:39.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:39.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:39.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:39.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:02.157944+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:40.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:24:41.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:24:41.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:24:41.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:24:41.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:24:41.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:24:41.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:24:41.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:24:41.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:02.157944+0000 2024-09-10T21:24:41.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:24:41.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-10T21:24:41.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:41.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-10T21:24:41.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-09-10T21:24:41.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:24:41.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:24:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:24:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:43.171 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-09-10T21:24:43.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:24:43.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:24:43.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:24:43.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:24:43.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:24:43.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:24:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:24:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:24:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:24:43.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:24:43.756 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:43.753+0000 7fa6028f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:43.758 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:43.755+0000 7fa6028f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:43.760 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:43.756+0000 7fa6028f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:24:44.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:44.873 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:44.870+0000 7fa6028f1640 -1 Falling back to public interface 2024-09-10T21:24:45.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:45.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:24:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:24:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:24:45.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:46.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:46.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:46.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:24:46.584 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:24:46.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:46.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:24:46.698 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:46.695+0000 7fa6028f1640 -1 osd.1 62 log_to_monitors true 2024-09-10T21:24:46.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:47.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:47.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:47.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:24:47.902 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:24:47.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:47.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:24:48.343 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [10,62) [v2:127.0.0.1:6810/2932166223,v1:127.0.0.1:6811/2932166223] [v2:127.0.0.1:6812/2932166223,v1:127.0.0.1:6813/2932166223] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:24:48.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:24:48.344 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:24:48.345 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:24:48.345 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:24:48.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:24:48.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:24:48.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:24:48.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:24:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:24:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:24:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:24:48.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:24:48.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:24:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-10T21:24:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-10T21:24:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-10T21:24:48.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:24:48.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:24:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841538 2024-09-10T21:24:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841538 2024-09-10T21:24:48.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-283467841538' 2024-09-10T21:24:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:24:48.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-10T21:24:48.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:24:48.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:24:48.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-10T21:24:48.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:24:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-10T21:24:48.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-10T21:24:48.962 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836507 2024-09-10T21:24:48.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:24:49.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-10T21:24:49.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:24:50.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:24:50.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:24:50.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-10T21:24:50.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:24:51.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:24:51.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:24:51.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-10T21:24:51.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:24:51.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-283467841538 2024-09-10T21:24:51.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:24:51.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:24:51.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-283467841538 2024-09-10T21:24:51.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:24:51.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841538 2024-09-10T21:24:51.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-09-10T21:24:51.811 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 283467841538 2024-09-10T21:24:51.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:24:52.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841538 -lt 283467841538 2024-09-10T21:24:52.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:24:52.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:24:52.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:24:52.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:24:52.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:24:52.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:24:52.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:24:52.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:24:52.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-10T21:24:53.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:24:53.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:24:53.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:24:53.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:24:53.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:24:53.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:24:53.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:24:55.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:24:55.664 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:24:55.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:24:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:24:55.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:55.691+0000 7f3f68e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:55.696 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:55.694+0000 7f3f68e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:55.697 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:55.695+0000 7f3f68e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:24:55.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:24:55.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:55.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:24:56.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:56.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:56.823+0000 7f3f68e0b640 -1 Falling back to public interface 2024-09-10T21:24:57.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:57.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:57.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:24:57.263 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:24:57.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:57.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:24:57.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:58.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:24:58.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:24:58.661+0000 7f3f68e0b640 -1 osd.0 68 log_to_monitors true 2024-09-10T21:24:58.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:24:59.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:24:59.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:24:59.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:24:59.858 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:24:59.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:24:59.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:00.138 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/2769719128,v1:127.0.0.1:6803/2769719128] [v2:127.0.0.1:6804/2769719128,v1:127.0.0.1:6805/2769719128] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:00.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:00.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:00.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:00.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:00.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:00.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:00.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:00.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:00.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:00.508 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:00.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:00.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:00.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:00.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-10T21:25:00.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-10T21:25:00.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-309237645314' 2024-09-10T21:25:00.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:00.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:00.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841541 2024-09-10T21:25:00.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841541 2024-09-10T21:25:00.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-309237645314 1-283467841541' 2024-09-10T21:25:00.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:00.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-309237645314 2024-09-10T21:25:00.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:00.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:00.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-309237645314 2024-09-10T21:25:00.717 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:00.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-10T21:25:00.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-09-10T21:25:00.718 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 309237645314 2024-09-10T21:25:00.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:00.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-10T21:25:00.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:00.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-283467841541 2024-09-10T21:25:00.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:00.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:00.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-283467841541 2024-09-10T21:25:00.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:00.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841541 2024-09-10T21:25:00.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 283467841541' 2024-09-10T21:25:00.998 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 283467841541 2024-09-10T21:25:00.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:01.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841541 -lt 283467841541 2024-09-10T21:25:01.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:01.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:01.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:01.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:01.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:01.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:01.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:01.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-09-10T21:25:02.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:02.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:25:02.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:02.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:02.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:02.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:02.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-09-10T21:25:03.978 INFO:tasks.workunit.client.0.smithi181.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-09-10T21:25:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:25:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:04.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:04.496 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:04.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:25:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:25:04.498 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:04.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:04.526 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:04.524+0000 7feb99bd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:04.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:04.526+0000 7feb99bd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:04.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:04.527+0000 7feb99bd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:04.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:25:04.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:04.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:25:04.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:04.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:04.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:04.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:04.781 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:04.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:04.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:05.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:05.646 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:05.644+0000 7feb99bd7640 -1 Falling back to public interface 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:06.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:06.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:07.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:07.281+0000 7feb99bd7640 -1 osd.1 73 log_to_monitors true 2024-09-10T21:25:07.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:07.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:07.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:25:07.362 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:25:07.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:07.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 77 up_thru 72 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6810/2607098917,v1:127.0.0.1:6811/2607098917] [v2:127.0.0.1:6812/2607098917,v1:127.0.0.1:6813/2607098917] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:07.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:07.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:07.707 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:07.707 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:07.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:07.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:07.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:07.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:07.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:07.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:07.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:07.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:07.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:08.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:08.115 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:08.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:08.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:08.115 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:08.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645316 2024-09-10T21:25:08.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645316 2024-09-10T21:25:08.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-309237645316' 2024-09-10T21:25:08.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:08.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:08.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481794 2024-09-10T21:25:08.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481794 2024-09-10T21:25:08.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-309237645316 1-330712481794' 2024-09-10T21:25:08.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:08.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-309237645316 2024-09-10T21:25:08.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:08.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:08.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-309237645316 2024-09-10T21:25:08.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:08.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645316 2024-09-10T21:25:08.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 309237645316' 2024-09-10T21:25:08.389 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 309237645316 2024-09-10T21:25:08.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:08.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645316 2024-09-10T21:25:08.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:09.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:25:09.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645316 -lt 309237645316 2024-09-10T21:25:10.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:10.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-330712481794 2024-09-10T21:25:10.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:10.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:10.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-330712481794 2024-09-10T21:25:10.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481794 2024-09-10T21:25:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-09-10T21:25:10.053 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 330712481794 2024-09-10T21:25:10.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481794 -lt 330712481794 2024-09-10T21:25:10.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:10.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:10.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:10.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:10.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:25:10.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:10.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:10.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:25:11.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:25:11.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:25:11.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:11.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:11.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:25:11.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:11.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:11.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:25:12.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:25:12.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:25:12.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:25:12.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:25:12.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:25:12.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:25:12.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:25:12.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:25:12.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:12.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:12.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:25:12.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:12.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:12.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:25:13.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:25:13.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:25:13.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:25:13.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T21:25:13.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:25:13.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T21:25:13.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:25:13.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:13.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:14.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:14.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:14.191 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:14.191 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:14.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:14.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:14.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:14.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:14.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-10T21:25:14.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-10T21:25:14.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:14.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:14.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:14.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-10T21:25:14.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:14.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:14.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:25:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:14.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-10T21:25:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:25:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:25:17.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:17.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:17.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:17.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:25:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:25:17.111 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:17.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:17.139 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:17.136+0000 7f10ca82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:17.141 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:17.139+0000 7f10ca82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:17.142 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:17.140+0000 7f10ca82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:17.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:17.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:17.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:18.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:18.269+0000 7f10ca82c640 -1 Falling back to public interface 2024-09-10T21:25:18.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:18.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:18.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:18.673 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:18.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:18.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:18.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:19.666 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:19.663+0000 7f10ca82c640 -1 osd.0 78 log_to_monitors true 2024-09-10T21:25:19.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:25:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:25:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:19.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 82 up_thru 77 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/351980118,v1:127.0.0.1:6803/351980118] [v2:127.0.0.1:6804/351980118,v1:127.0.0.1:6805/351980118] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:20.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:20.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:20.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:20.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:20.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:20.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-10T21:25:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-10T21:25:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-352187318274' 2024-09-10T21:25:21.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:21.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:21.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481797 2024-09-10T21:25:21.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481797 2024-09-10T21:25:21.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-09-10T21:25:21.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:21.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-352187318274 2024-09-10T21:25:21.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:21.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:21.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-352187318274 2024-09-10T21:25:21.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:21.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-10T21:25:21.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-09-10T21:25:21.181 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 352187318274 2024-09-10T21:25:21.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:21.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 352187318274 2024-09-10T21:25:21.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:22.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:25:22.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:22.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 352187318274 2024-09-10T21:25:22.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:23.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:25:23.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:24.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-10T21:25:24.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:24.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-330712481797 2024-09-10T21:25:24.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:24.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:24.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-330712481797 2024-09-10T21:25:24.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:24.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481797 2024-09-10T21:25:24.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-09-10T21:25:24.060 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 330712481797 2024-09-10T21:25:24.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:24.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481798 -lt 330712481797 2024-09-10T21:25:24.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:24.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:24.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:24.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:24.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:24.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:24.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:24.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:24.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:24.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:25.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-10T21:25:25.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:25.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:25.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:25.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:27.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:27.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:25:27.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:27.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:27.665 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:27.662+0000 7fdf32a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:27.667 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:27.665+0000 7fdf32a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:27.669 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:27.667+0000 7fdf32a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:27.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:27.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:28.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:28.785 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:28.783+0000 7fdf32a1a640 -1 Falling back to public interface 2024-09-10T21:25:29.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:29.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:29.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:29.202 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:29.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:29.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:29.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:30.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:30.330+0000 7fdf32a1a640 -1 osd.1 84 log_to_monitors true 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:30.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:30.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:31.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:25:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:25:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:31.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/3257825899,v1:127.0.0.1:6811/3257825899] [v2:127.0.0.1:6812/3257825899,v1:127.0.0.1:6813/3257825899] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:25:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:32.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:32.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:32.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:32.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:32.496 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:32.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:32.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:32.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:32.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318277 2024-09-10T21:25:32.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318277 2024-09-10T21:25:32.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-352187318277' 2024-09-10T21:25:32.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:32.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:32.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122050 2024-09-10T21:25:32.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122050 2024-09-10T21:25:32.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-09-10T21:25:32.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:32.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-352187318277 2024-09-10T21:25:32.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:32.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:32.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-352187318277 2024-09-10T21:25:32.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:32.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318277 2024-09-10T21:25:32.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-09-10T21:25:32.707 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 352187318277 2024-09-10T21:25:32.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318277 -lt 352187318277 2024-09-10T21:25:32.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:32.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-377957122050 2024-09-10T21:25:32.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:32.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:32.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-377957122050 2024-09-10T21:25:32.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:32.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122050 2024-09-10T21:25:32.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-09-10T21:25:32.986 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 377957122050 2024-09-10T21:25:32.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:33.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122050 -lt 377957122050 2024-09-10T21:25:33.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:33.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:33.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:33.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:33.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:33.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:33.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:33.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:33.879 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:33.879 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:33.879 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:34.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:34.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:34.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:34.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:34.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-10T21:25:34.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:25:34.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:34.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:34.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:34.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-10T21:25:34.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:34.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:34.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:25:34.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:34.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:34.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:36.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:36.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:36.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:25:36.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:25:36.342 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:25:36.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:36.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:25:36.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:25:36.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:36.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:36.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:36.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:36.367+0000 7f584bc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:36.373 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:36.370+0000 7f584bc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:36.374 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:36.371+0000 7f584bc1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:25:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:36.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:37.248 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:37.244+0000 7f584bc1a640 -1 Falling back to public interface 2024-09-10T21:25:37.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:37.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:37.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:37.913 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:37.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:37.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:38.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:38.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:38.763+0000 7f584bc1a640 -1 osd.0 89 log_to_monitors true 2024-09-10T21:25:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:25:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:25:39.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:39.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:39.493 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 93 up_thru 88 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/2723819042,v1:127.0.0.1:6803/2723819042] [v2:127.0.0.1:6804/2723819042,v1:127.0.0.1:6805/2723819042] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:39.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:39.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:39.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:39.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:39.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:39.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:39.989 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:39.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:39.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:39.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:40.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958530 2024-09-10T21:25:40.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958530 2024-09-10T21:25:40.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-399431958530' 2024-09-10T21:25:40.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:40.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122052 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122052 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-399431958530 1-377957122052' 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-399431958530 2024-09-10T21:25:40.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:40.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:40.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-399431958530 2024-09-10T21:25:40.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:40.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958530 2024-09-10T21:25:40.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-09-10T21:25:40.371 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 399431958530 2024-09-10T21:25:40.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:40.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318277 -lt 399431958530 2024-09-10T21:25:40.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:41.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:25:41.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:41.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958530 -lt 399431958530 2024-09-10T21:25:41.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:41.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-377957122052 2024-09-10T21:25:41.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:41.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:41.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-377957122052 2024-09-10T21:25:41.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:41.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122052 2024-09-10T21:25:41.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 377957122052' 2024-09-10T21:25:41.958 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 377957122052 2024-09-10T21:25:41.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122053 -lt 377957122052 2024-09-10T21:25:42.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:42.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:42.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:42.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:42.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:42.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:42.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:42.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:42.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:42.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:42.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:42.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:42.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:42.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:42.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:43.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:43.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:43.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:43.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-09-10T21:25:43.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:43.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-10T21:25:43.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:43.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:43.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:43.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:43.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:45.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:45.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:45.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:45.764+0000 7fb870214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:45.769 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:45.767+0000 7fb870214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:45.770 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:45.768+0000 7fb870214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:25:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:25:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:46.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:46.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:46.053 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:46.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:46.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:46.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:47.108 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:47.106+0000 7fb870214640 -1 Falling back to public interface 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:47.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:47.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:48.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:48.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:48.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:25:48.616 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:25:48.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:48.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:48.724 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:48.721+0000 7fb870214640 -1 osd.1 95 log_to_monitors true 2024-09-10T21:25:49.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:50.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:25:50.397 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/3214935734,v1:127.0.0.1:6811/3214935734] [v2:127.0.0.1:6812/3214935734,v1:127.0.0.1:6813/3214935734] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:25:50.398 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:25:50.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:25:50.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:25:50.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:25:50.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:25:50.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:25:50.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:25:50.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:25:50.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:25:50.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:25:50.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:50.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:25:50.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958533 2024-09-10T21:25:50.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958533 2024-09-10T21:25:50.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-399431958533' 2024-09-10T21:25:50.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:25:50.868 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:25:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-10T21:25:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-10T21:25:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-09-10T21:25:50.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-399431958533 2024-09-10T21:25:50.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:50.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:25:50.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-399431958533 2024-09-10T21:25:50.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958533 2024-09-10T21:25:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-09-10T21:25:50.975 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 399431958533 2024-09-10T21:25:50.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:51.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958532 -lt 399431958533 2024-09-10T21:25:51.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:25:52.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:52.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958532 -lt 399431958533 2024-09-10T21:25:52.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:25:53.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:25:53.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:25:53.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958533 -lt 399431958533 2024-09-10T21:25:53.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:25:53.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-425201762306 2024-09-10T21:25:53.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:25:53.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:25:53.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-425201762306 2024-09-10T21:25:53.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:25:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-10T21:25:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-09-10T21:25:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 425201762306 2024-09-10T21:25:53.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:25:54.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762306 -lt 425201762306 2024-09-10T21:25:54.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:25:54.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:54.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:54.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:25:54.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:25:54.470 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:25:54.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:25:54.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:25:54.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:25:54.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:25:55.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-10T21:25:55.088 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-09-10T21:25:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:25:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:25:55.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-09-10T21:25:55.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-09-10T21:25:55.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-10T21:25:55.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:25:55.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:25:55.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:25:55.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-10T21:25:57.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:25:57.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:25:57.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:25:57.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:25:57.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:25:57.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:25:57.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:25:57.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:25:57.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:25:57.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:25:57.598 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:57.596+0000 7fd733a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:57.601 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:57.598+0000 7fd733a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:57.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:57.599+0000 7fd733a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:57.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:25:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:25:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:57.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:58.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:25:58.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:25:58.956+0000 7fd733a0d640 -1 Falling back to public interface 2024-09-10T21:25:59.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:25:59.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:25:59.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:25:59.149 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:25:59.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:25:59.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:25:59.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:00.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:00.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:00.430 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:00.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:00.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:00.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:00.692+0000 7fd733a0d640 -1 osd.0 101 log_to_monitors true 2024-09-10T21:26:00.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:01.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:01.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:01.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:26:01.714 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:26:01.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:01.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/1642504909,v1:127.0.0.1:6803/1642504909] [v2:127.0.0.1:6804/1642504909,v1:127.0.0.1:6805/1642504909] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:26:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:02.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:02.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:02.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:02.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:02.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:02.081 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:02.082 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:02.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:02.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:02.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:02.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:02.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:02.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:02.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:02.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:02.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:02.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:02.448 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:02.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:02.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:02.448 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:02.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-10T21:26:02.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-10T21:26:02.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566082' 2024-09-10T21:26:02.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:02.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:02.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762309 2024-09-10T21:26:02.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762309 2024-09-10T21:26:02.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-09-10T21:26:02.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:02.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-450971566082 2024-09-10T21:26:02.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:02.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:02.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-450971566082 2024-09-10T21:26:02.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-10T21:26:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-09-10T21:26:02.657 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 450971566082 2024-09-10T21:26:02.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566082 -lt 450971566082 2024-09-10T21:26:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:02.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-425201762309 2024-09-10T21:26:02.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:02.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:02.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-425201762309 2024-09-10T21:26:02.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:02.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762309 2024-09-10T21:26:02.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-09-10T21:26:02.937 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 425201762309 2024-09-10T21:26:02.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:03.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762309 -lt 425201762309 2024-09-10T21:26:03.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:03.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:03.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:03.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:03.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:03.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:03.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:03.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:03.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:04.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:04.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:26:04.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:04.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:04.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:06.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:26:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:26:06.252 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:06.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:06.279 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:06.277+0000 7f8bc9e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:06.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:06.279+0000 7f8bc9e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:06.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:06.281+0000 7f8bc9e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:06.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:06.532 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:06.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:06.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:06.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:07.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:07.397+0000 7f8bc9e11640 -1 Falling back to public interface 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:08.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:08.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:08.716+0000 7f8bc9e11640 -1 osd.1 106 log_to_monitors true 2024-09-10T21:26:09.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:09.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:09.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:09.134 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:09.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:09.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3343320308,v1:127.0.0.1:6811/3343320308] [v2:127.0.0.1:6812/3343320308,v1:127.0.0.1:6813/3343320308] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:09.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:09.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:09.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:09.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:09.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:09.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:09.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:09.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:10.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:10.025 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:10.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:10.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:10.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:10.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566084 2024-09-10T21:26:10.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566084 2024-09-10T21:26:10.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566084' 2024-09-10T21:26:10.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:10.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:10.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=472446402562 2024-09-10T21:26:10.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 472446402562 2024-09-10T21:26:10.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566084 1-472446402562' 2024-09-10T21:26:10.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:10.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-450971566084 2024-09-10T21:26:10.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:10.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:10.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-450971566084 2024-09-10T21:26:10.362 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:10.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566084 2024-09-10T21:26:10.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 450971566084' 2024-09-10T21:26:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 450971566084 2024-09-10T21:26:10.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:10.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566083 -lt 450971566084 2024-09-10T21:26:10.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:26:11.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:26:11.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:11.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566084 2024-09-10T21:26:11.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:11.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-472446402562 2024-09-10T21:26:11.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:11.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-472446402562 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=472446402562 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 472446402562 2024-09-10T21:26:11.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:12.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 472446402562 -lt 472446402562 2024-09-10T21:26:12.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:12.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:12.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:12.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:12.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:12.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:12.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:12.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:12.561 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:12.561 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:12.561 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:12.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:26:12.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:12.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:12.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:13.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-10T21:26:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T21:26:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:26:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T21:26:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:26:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:26:13.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:26:13.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:13.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:13.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:13.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:13.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:13.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:13.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:13.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-10T21:26:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:14.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:26:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:14.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-10T21:26:16.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:26:16.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:16.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:16.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:26:16.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:16.433 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:16.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:26:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:26:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:26:16.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:16.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:26:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:26:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:16.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:16.463 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:16.461+0000 7fcc06a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:16.466 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:16.464+0000 7fcc06a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:16.467 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:16.465+0000 7fcc06a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:16.719 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:16.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:16.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:16.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:17.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:18.085 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:18.083+0000 7fcc06a2e640 -1 Falling back to public interface 2024-09-10T21:26:18.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:19.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:19.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:19.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:19.655 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:19.653+0000 7fcc06a2e640 -1 osd.1 112 log_to_monitors true 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:20.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1085933060,v1:127.0.0.1:6811/1085933060] [v2:127.0.0.1:6812/1085933060,v1:127.0.0.1:6813/1085933060] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:20.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:20.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:20.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:20.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:20.912 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:20.913 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:20.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:20.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:20.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:21.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:21.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:21.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:21.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:21.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:21.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:21.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:21.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566088 2024-09-10T21:26:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566088 2024-09-10T21:26:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566088' 2024-09-10T21:26:21.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:21.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:21.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=498216206338 2024-09-10T21:26:21.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 498216206338 2024-09-10T21:26:21.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-09-10T21:26:21.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:21.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-450971566088 2024-09-10T21:26:21.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:21.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:21.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-450971566088 2024-09-10T21:26:21.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:21.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566088 2024-09-10T21:26:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-09-10T21:26:21.654 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 450971566088 2024-09-10T21:26:21.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566086 -lt 450971566088 2024-09-10T21:26:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:26:22.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:26:22.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566088 -lt 450971566088 2024-09-10T21:26:23.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:23.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-498216206338 2024-09-10T21:26:23.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:23.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:23.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-498216206338 2024-09-10T21:26:23.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=498216206338 2024-09-10T21:26:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-09-10T21:26:23.258 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 498216206338 2024-09-10T21:26:23.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:23.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-10T21:26:23.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:23.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:23.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:23.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:23.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:23.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:24.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:24.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:24.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:24.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:24.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-10T21:26:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:24.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-10T21:26:24.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:26:24.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:24.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:26.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:26.770 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:26.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:26:26.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:26:26.772 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:26.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:26.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:26.797+0000 7fe457842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:26.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:26.800+0000 7fe457842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:26.805 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:26.802+0000 7fe457842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:27.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:27.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:27.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:28.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:28.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:28.426+0000 7fe457842640 -1 Falling back to public interface 2024-09-10T21:26:28.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:29.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:29.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:29.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:29.673+0000 7fe457842640 -1 osd.1 117 log_to_monitors true 2024-09-10T21:26:29.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:30.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:26:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:26:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:30.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/1335384220,v1:127.0.0.1:6811/1335384220] [v2:127.0.0.1:6812/1335384220,v1:127.0.0.1:6813/1335384220] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:31.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:31.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:31.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:31.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:31.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:31.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:31.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:31.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:31.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:31.735 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:31.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:31.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:31.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:31.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566091 2024-09-10T21:26:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566091 2024-09-10T21:26:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566091' 2024-09-10T21:26:31.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:31.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-450971566091 2024-09-10T21:26:31.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:31.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:31.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-450971566091 2024-09-10T21:26:31.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566091 2024-09-10T21:26:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-09-10T21:26:31.945 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 450971566091 2024-09-10T21:26:31.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566091 2024-09-10T21:26:32.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:26:33.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:26:33.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:33.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566091 -lt 450971566091 2024-09-10T21:26:33.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:33.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-519691042818 2024-09-10T21:26:33.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:33.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:33.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-519691042818 2024-09-10T21:26:33.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:33.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-10T21:26:33.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-09-10T21:26:33.513 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 519691042818 2024-09-10T21:26:33.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:33.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042818 -lt 519691042818 2024-09-10T21:26:33.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:33.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:33.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:34.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:34.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:34.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:34.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:34.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:34.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:34.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:34.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:34.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:34.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:34.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-09-10T21:26:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-10T21:26:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:26:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:34.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:26:35.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:35.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:35.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:26:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:26:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:26:37.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:37.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:37.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:37.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:37.462 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:37.458+0000 7fd01a024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:37.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:37.461+0000 7fd01a024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:37.464 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:37.462+0000 7fd01a024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:37.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:38.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:38.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:38.576+0000 7fd01a024640 -1 Falling back to public interface 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:39.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:39.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:40.307 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:40.305+0000 7fd01a024640 -1 osd.1 123 log_to_monitors true 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:40.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:40.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:41.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:26:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:26:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:41.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/3962408690,v1:127.0.0.1:6811/3962408690] [v2:127.0.0.1:6812/3962408690,v1:127.0.0.1:6813/3962408690] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:42.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:42.029 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:42.029 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:42.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:42.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:42.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:42.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:42.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:42.392 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:42.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:42.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:42.393 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:42.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566094 2024-09-10T21:26:42.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566094 2024-09-10T21:26:42.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566094' 2024-09-10T21:26:42.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:42.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:42.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=545460846594 2024-09-10T21:26:42.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 545460846594 2024-09-10T21:26:42.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-09-10T21:26:42.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:42.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-450971566094 2024-09-10T21:26:42.598 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:42.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:42.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-450971566094 2024-09-10T21:26:42.599 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566094 2024-09-10T21:26:42.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-09-10T21:26:42.601 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 450971566094 2024-09-10T21:26:42.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566092 -lt 450971566094 2024-09-10T21:26:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:26:43.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:26:43.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:44.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566094 -lt 450971566094 2024-09-10T21:26:44.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:44.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-545460846594 2024-09-10T21:26:44.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:44.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:44.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-545460846594 2024-09-10T21:26:44.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:44.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=545460846594 2024-09-10T21:26:44.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-09-10T21:26:44.162 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 545460846594 2024-09-10T21:26:44.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:44.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 545460846594 -lt 545460846594 2024-09-10T21:26:44.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:44.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:44.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:44.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:44.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:44.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:45.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:45.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:45.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:45.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:45.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:45.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:45.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:45.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-10T21:26:45.395 INFO:tasks.workunit.client.0.smithi181.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-10T21:26:45.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:45.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-10T21:26:45.395 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-09-10T21:26:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:26:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-10T21:26:45.397 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-09-10T21:26:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:26:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-10T21:26:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-09-10T21:26:45.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:26:45.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:45.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:45.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-10T21:26:45.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-09-10T21:26:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:26:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:26:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-10T21:26:45.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:45.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:26:45.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-10T21:26:47.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:47.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:47.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:47.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:47.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:47.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:47.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:47.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:26:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:26:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:47.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:47.679 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:47.677+0000 7f395d632640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:47.681 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:47.679+0000 7f395d632640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:47.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:47.681+0000 7f395d632640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:47.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:26:47.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:47.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:26:47.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:47.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:47.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:48.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:48.800 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:48.798+0000 7f395d632640 -1 Falling back to public interface 2024-09-10T21:26:49.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:49.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:49.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:50.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:50.600 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:50.598+0000 7f395d632640 -1 osd.0 129 log_to_monitors true 2024-09-10T21:26:50.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:51.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:51.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:51.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:26:51.866 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:26:51.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:51.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/487011403,v1:127.0.0.1:6803/487011403] [v2:127.0.0.1:6804/487011403,v1:127.0.0.1:6805/487011403] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:26:52.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:26:52.262 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:26:52.262 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:26:52.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:26:52.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:26:52.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:26:52.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:26:52.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:26:52.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:26:52.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:26:52.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:26:52.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:26:52.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:26:52.629 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:26:52.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:26:52.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:52.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:26:52.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=571230650370 2024-09-10T21:26:52.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 571230650370 2024-09-10T21:26:52.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650370' 2024-09-10T21:26:52.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:26:52.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:26:52.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=545460846597 2024-09-10T21:26:52.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 545460846597 2024-09-10T21:26:52.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-09-10T21:26:52.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:52.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-571230650370 2024-09-10T21:26:52.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:52.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:26:52.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-571230650370 2024-09-10T21:26:52.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:52.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=571230650370 2024-09-10T21:26:52.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-09-10T21:26:52.839 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 571230650370 2024-09-10T21:26:52.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:26:53.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650370 -lt 571230650370 2024-09-10T21:26:53.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:26:53.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-545460846597 2024-09-10T21:26:53.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:26:53.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:26:53.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-545460846597 2024-09-10T21:26:53.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:26:53.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=545460846597 2024-09-10T21:26:53.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-09-10T21:26:53.124 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 545460846597 2024-09-10T21:26:53.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:26:53.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 545460846597 -lt 545460846597 2024-09-10T21:26:53.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:26:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:53.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:26:53.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:26:54.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:26:54.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:26:54.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:26:54.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:26:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:26:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:26:54.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:26:54.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:26:54.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:26:54.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:26:54.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-09-10T21:26:56.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:26:56.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:26:56.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:26:56.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:26:56.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:26:56.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:26:56.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:26:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:26:56.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:26:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:56.600+0000 7f7971ced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:56.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:56.603+0000 7f7971ced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:56.606 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:56.604+0000 7f7971ced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:26:56.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:56.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:57.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:58.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:26:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:26:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:58.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:58.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:58.204+0000 7f7971ced640 -1 Falling back to public interface 2024-09-10T21:26:58.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:26:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:26:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:26:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:26:59.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:26:59.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:26:59.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:26:59.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:26:59.746+0000 7f7971ced640 -1 osd.1 134 log_to_monitors true 2024-09-10T21:27:00.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:00.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:00.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:27:00.705 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:27:00.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:00.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/936777332,v1:127.0.0.1:6811/936777332] [v2:127.0.0.1:6812/936777332,v1:127.0.0.1:6813/936777332] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:01.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:01.144 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:01.144 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:01.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:01.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:01.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:01.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:01.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:01.512 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:01.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:01.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:01.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=571230650373 2024-09-10T21:27:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 571230650373 2024-09-10T21:27:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650373' 2024-09-10T21:27:01.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:01.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:01.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=592705486850 2024-09-10T21:27:01.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 592705486850 2024-09-10T21:27:01.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-09-10T21:27:01.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:01.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-571230650373 2024-09-10T21:27:01.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:01.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:01.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-571230650373 2024-09-10T21:27:01.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:01.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=571230650373 2024-09-10T21:27:01.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-09-10T21:27:01.721 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 571230650373 2024-09-10T21:27:01.722 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650371 -lt 571230650373 2024-09-10T21:27:01.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:02.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:27:02.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:03.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650373 -lt 571230650373 2024-09-10T21:27:03.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:03.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-592705486850 2024-09-10T21:27:03.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:03.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:03.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-592705486850 2024-09-10T21:27:03.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=592705486850 2024-09-10T21:27:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-09-10T21:27:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 592705486850 2024-09-10T21:27:03.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:03.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 592705486850 -lt 592705486850 2024-09-10T21:27:03.561 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:03.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:03.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:03.938 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:04.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:04.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:04.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:04.208 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:27:04.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-10T21:27:04.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-09-10T21:27:04.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:27:04.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:27:04.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-10T21:27:04.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:27:04.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:04.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:04.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:04.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-10T21:27:04.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:04.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:04.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:27:04.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:04.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:04.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-09-10T21:27:06.395 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:27:06.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:27:06.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:06.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:06.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:06.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:06.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:06.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:06.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:06.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:06.944 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:06.942+0000 7fb343033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:06.946 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:06.944+0000 7fb343033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:06.947 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:06.945+0000 7fb343033640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:07.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:07.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:08.305 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:08.303+0000 7fb343033640 -1 Falling back to public interface 2024-09-10T21:27:08.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:08.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:08.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:08.481 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:08.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:08.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:08.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:09.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:09.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:09.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:09.772 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:09.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:09.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:09.843 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:09.840+0000 7fb343033640 -1 osd.1 140 log_to_monitors true 2024-09-10T21:27:10.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:11.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:11.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:11.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:27:11.157 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:27:11.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:11.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/2073779201,v1:127.0.0.1:6811/2073779201] [v2:127.0.0.1:6812/2073779201,v1:127.0.0.1:6813/2073779201] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:11.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:11.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:11.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:11.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:11.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:11.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:11.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:11.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:11.817 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:11.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:11.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:11.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=571230650376 2024-09-10T21:27:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 571230650376 2024-09-10T21:27:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650376' 2024-09-10T21:27:11.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:11.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290626 2024-09-10T21:27:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290626 2024-09-10T21:27:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-09-10T21:27:12.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:12.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-571230650376 2024-09-10T21:27:12.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:12.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:12.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-571230650376 2024-09-10T21:27:12.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=571230650376 2024-09-10T21:27:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-09-10T21:27:12.059 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 571230650376 2024-09-10T21:27:12.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:12.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650374 -lt 571230650376 2024-09-10T21:27:12.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:13.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:27:13.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:13.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650376 -lt 571230650376 2024-09-10T21:27:13.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:13.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-618475290626 2024-09-10T21:27:13.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:13.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:13.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-618475290626 2024-09-10T21:27:13.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:13.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290626 2024-09-10T21:27:13.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-09-10T21:27:13.623 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 618475290626 2024-09-10T21:27:13.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:13.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290626 -lt 618475290626 2024-09-10T21:27:13.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:13.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:13.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:14.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:14.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:14.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:14.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:14.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:14.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:14.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:14.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:14.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:14.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:14.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:27:14.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-10T21:27:14.898 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-10T21:27:14.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:15.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:15.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-10T21:27:15.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:15.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:15.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:27:15.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:15.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:15.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-09-10T21:27:16.728 INFO:tasks.workunit.client.0.smithi181.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:17.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:17.497 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:17.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:27:17.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:27:17.499 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:17.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:17.528 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:17.525+0000 7fad1cbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:17.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:17.527+0000 7fad1cbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:17.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:17.528+0000 7fad1cbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:17.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:18.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:19.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:19.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:19.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:19.060 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:19.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:19.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:19.136 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:19.133+0000 7fad1cbae640 -1 Falling back to public interface 2024-09-10T21:27:19.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:20.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:20.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:20.452+0000 7fad1cbae640 -1 osd.0 145 log_to_monitors true 2024-09-10T21:27:20.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:21.995 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/704837149,v1:127.0.0.1:6803/704837149] [v2:127.0.0.1:6804/704837149,v1:127.0.0.1:6805/704837149] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:27:21.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:21.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:21.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:21.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:21.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:21.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:21.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:22.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:22.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:22.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:22.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:22.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:22.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:22.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:22.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:22.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=639950127106 2024-09-10T21:27:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 639950127106 2024-09-10T21:27:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127106' 2024-09-10T21:27:22.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:22.468 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:22.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290629 2024-09-10T21:27:22.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290629 2024-09-10T21:27:22.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-09-10T21:27:22.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:22.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-639950127106 2024-09-10T21:27:22.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:22.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:22.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-639950127106 2024-09-10T21:27:22.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:22.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=639950127106 2024-09-10T21:27:22.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-09-10T21:27:22.573 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 639950127106 2024-09-10T21:27:22.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:22.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 639950127106 2024-09-10T21:27:22.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:23.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:27:23.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:24.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127106 -lt 639950127106 2024-09-10T21:27:24.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:24.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-618475290629 2024-09-10T21:27:24.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:24.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:24.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-618475290629 2024-09-10T21:27:24.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:24.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290629 2024-09-10T21:27:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-09-10T21:27:24.137 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 618475290629 2024-09-10T21:27:24.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:24.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290629 -lt 618475290629 2024-09-10T21:27:24.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:24.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:24.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:24.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:24.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:24.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:24.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:24.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:24.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:24.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:24.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:25.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:25.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:25.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:25.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-10T21:27:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:25.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:25.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:27:25.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:25.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:25.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-10T21:27:27.053 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:27:27.053 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:27.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:27:27.576 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:27:27.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:27.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:27:27.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:27:27.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:27.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:27.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:27.605 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:27.603+0000 7fe8c9d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:27.608 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:27.606+0000 7fe8c9d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:27.609 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:27.607+0000 7fe8c9d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:27.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:27.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:27.874 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:27.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:27.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:28.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:28.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:28.720+0000 7fe8c9d02640 -1 Falling back to public interface 2024-09-10T21:27:29.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:29.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:29.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:29.153 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:29.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:29.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:29.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:30.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:30.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:30.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:30.438 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:30.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:30.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:30.459 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:30.457+0000 7fe8c9d02640 -1 osd.1 150 log_to_monitors true 2024-09-10T21:27:30.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:31.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:31.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:31.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:27:31.747 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:27:31.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:31.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/3522335023,v1:127.0.0.1:6811/3522335023] [v2:127.0.0.1:6812/3522335023,v1:127.0.0.1:6813/3522335023] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:32.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:32.063 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:32.063 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:32.155 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:32.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:32.432 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:32.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:32.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:32.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:32.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=639950127109 2024-09-10T21:27:32.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 639950127109 2024-09-10T21:27:32.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127109' 2024-09-10T21:27:32.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:32.536 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:32.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=661424963586 2024-09-10T21:27:32.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 661424963586 2024-09-10T21:27:32.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-09-10T21:27:32.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:32.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-639950127109 2024-09-10T21:27:32.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:32.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-639950127109 2024-09-10T21:27:32.641 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=639950127109 2024-09-10T21:27:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-09-10T21:27:32.642 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 639950127109 2024-09-10T21:27:32.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127108 -lt 639950127109 2024-09-10T21:27:32.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:33.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:27:33.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:34.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127109 -lt 639950127109 2024-09-10T21:27:34.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:34.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-661424963586 2024-09-10T21:27:34.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:34.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:34.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-661424963586 2024-09-10T21:27:34.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:34.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=661424963586 2024-09-10T21:27:34.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-09-10T21:27:34.216 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 661424963586 2024-09-10T21:27:34.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:34.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963586 -lt 661424963586 2024-09-10T21:27:34.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:34.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:34.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:34.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:34.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:35.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:35.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:35.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:35.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:35.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:35.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:35.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:35.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:27:35.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-10T21:27:35.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:35.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:35.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:35.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:37.166 INFO:tasks.workunit.client.0.smithi181.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:37.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:37.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:27:37.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:37.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:37.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:37.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:37.800+0000 7fb1e9211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:37.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:37.803+0000 7fb1e9211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:37.806 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:37.804+0000 7fb1e9211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:38.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:38.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:38.063 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:38.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:38.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:38.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:38.909 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:38.907+0000 7fb1e9211640 -1 Falling back to public interface 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:39.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:39.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:40.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:40.455+0000 7fb1e9211640 -1 osd.0 156 log_to_monitors true 2024-09-10T21:27:40.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:40.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:41.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:42.276 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/2948457193,v1:127.0.0.1:6803/2948457193] [v2:127.0.0.1:6804/2948457193,v1:127.0.0.1:6805/2948457193] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:42.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:42.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:42.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:42.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:42.742 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:42.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:42.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:42.743 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:42.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=687194767362 2024-09-10T21:27:42.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 687194767362 2024-09-10T21:27:42.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767362' 2024-09-10T21:27:42.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:42.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:42.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=661424963589 2024-09-10T21:27:42.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 661424963589 2024-09-10T21:27:42.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-09-10T21:27:42.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:42.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-687194767362 2024-09-10T21:27:42.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:42.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:42.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-687194767362 2024-09-10T21:27:42.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:42.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=687194767362 2024-09-10T21:27:42.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-09-10T21:27:42.952 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 687194767362 2024-09-10T21:27:42.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:43.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767362 -lt 687194767362 2024-09-10T21:27:43.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:43.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-661424963589 2024-09-10T21:27:43.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:43.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:43.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-661424963589 2024-09-10T21:27:43.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:43.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=661424963589 2024-09-10T21:27:43.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-09-10T21:27:43.232 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 661424963589 2024-09-10T21:27:43.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:43.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963588 -lt 661424963589 2024-09-10T21:27:43.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:44.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:27:44.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:44.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963588 -lt 661424963589 2024-09-10T21:27:44.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:27:45.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:27:45.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:46.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963589 -lt 661424963589 2024-09-10T21:27:46.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:46.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:46.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:46.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:46.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:46.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:46.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:46.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:46.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:46.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:46.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:46.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:46.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:46.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:46.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:47.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:27:47.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:47.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:47.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-10T21:27:48.325 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:27:48.325 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:48.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:48.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:48.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:48.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:27:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:27:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:48.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:48.934 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:48.932+0000 7fee220fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:48.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:48.934+0000 7fee220fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:48.936+0000 7fee220fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:49.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:50.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:50.078+0000 7fee220fe640 -1 Falling back to public interface 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:50.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:50.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:51.227 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:51.225+0000 7fee220fe640 -1 osd.1 162 log_to_monitors true 2024-09-10T21:27:51.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:51.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 166 up_thru 166 down_at 163 last_clean_interval [154,162) [v2:127.0.0.1:6810/2690095693,v1:127.0.0.1:6811/2690095693] [v2:127.0.0.1:6812/2690095693,v1:127.0.0.1:6813/2690095693] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:27:52.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:27:52.119 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:27:52.119 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:27:52.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:27:52.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:27:52.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:27:52.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:27:52.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:27:52.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:27:52.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:27:52.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:27:52.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:27:52.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:27:52.492 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:27:52.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:27:52.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:52.492 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:27:52.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=687194767365 2024-09-10T21:27:52.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 687194767365 2024-09-10T21:27:52.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767365' 2024-09-10T21:27:52.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:27:52.691 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:27:52.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=712964571138 2024-09-10T21:27:52.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 712964571138 2024-09-10T21:27:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767365 1-712964571138' 2024-09-10T21:27:52.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:52.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-687194767365 2024-09-10T21:27:52.794 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:52.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:27:52.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-687194767365 2024-09-10T21:27:52.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=687194767365 2024-09-10T21:27:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-09-10T21:27:52.797 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 687194767365 2024-09-10T21:27:52.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:27:53.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767365 -lt 687194767365 2024-09-10T21:27:53.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:27:53.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-712964571138 2024-09-10T21:27:53.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:27:53.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:27:53.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-712964571138 2024-09-10T21:27:53.077 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:27:53.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=712964571138 2024-09-10T21:27:53.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 712964571138' 2024-09-10T21:27:53.079 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 712964571138 2024-09-10T21:27:53.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:27:53.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 712964571138 -lt 712964571138 2024-09-10T21:27:53.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:27:53.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:53.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:53.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:27:53.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:27:53.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:27:53.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:27:53.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:27:53.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:27:53.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:27:53.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:27:53.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:27:53.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:27:53.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:27:53.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:27:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:27:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:27:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:27:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:27:54.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-10T21:27:54.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:27:54.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:27:54.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:27:54.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:54.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:27:56.206 INFO:tasks.workunit.client.0.smithi181.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:27:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:27:56.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:27:56.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:27:56.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:27:56.813 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:56.811+0000 7ffafdc28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:56.816 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:56.814+0000 7ffafdc28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:56.817 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:56.815+0000 7ffafdc28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:27:57.078 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:27:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:57.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:57.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:58.186 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:27:58.184+0000 7ffafdc28640 -1 Falling back to public interface 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:58.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:27:58.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:27:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:00.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:00.309 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:00.307+0000 7ffafdc28640 -1 osd.0 168 log_to_monitors true 2024-09-10T21:28:01.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:01.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:01.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:01.053 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:01.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:01.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:01.579 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 172 up_thru 172 down_at 169 last_clean_interval [160,168) [v2:127.0.0.1:6802/3886174285,v1:127.0.0.1:6803/3886174285] [v2:127.0.0.1:6804/3886174285,v1:127.0.0.1:6805/3886174285] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:28:01.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:01.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:01.581 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:01.581 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:01.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:01.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:01.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:01.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:01.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:01.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:01.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:01.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:01.674 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:01.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:01.953 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:01.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:01.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:01.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:28:02.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374914 2024-09-10T21:28:02.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374914 2024-09-10T21:28:02.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374914' 2024-09-10T21:28:02.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:02.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:28:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=712964571141 2024-09-10T21:28:02.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 712964571141 2024-09-10T21:28:02.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374914 1-712964571141' 2024-09-10T21:28:02.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:02.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374914 2024-09-10T21:28:02.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:02.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:28:02.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374914 2024-09-10T21:28:02.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:02.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374914 2024-09-10T21:28:02.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374914' 2024-09-10T21:28:02.189 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 738734374914 2024-09-10T21:28:02.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:02.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 738734374914 2024-09-10T21:28:02.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:28:03.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:28:03.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:03.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374914 -lt 738734374914 2024-09-10T21:28:03.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:03.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-712964571141 2024-09-10T21:28:03.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:03.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:28:03.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-712964571141 2024-09-10T21:28:03.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:03.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=712964571141 2024-09-10T21:28:03.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 712964571141' 2024-09-10T21:28:03.753 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 712964571141 2024-09-10T21:28:03.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:28:04.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 712964571141 -lt 712964571141 2024-09-10T21:28:04.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:28:04.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:04.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:04.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:28:04.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:28:04.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:28:04.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:28:04.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:04.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:05.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:28:05.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:28:05.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:28:05.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:05.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-10T21:28:06.630 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:28:06.630 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:28:07.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:28:07.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:28:07.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:28:07.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:28:07.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:28:07.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:28:07.201 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:07.198+0000 7f688840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:07.203 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:07.200+0000 7f688840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:07.205 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:07.202+0000 7f688840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:07.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:07.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:08.559+0000 7f688840d640 -1 Falling back to public interface 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:08.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:09.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:10.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:10.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:10.499 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:10.496+0000 7f688840d640 -1 osd.1 174 log_to_monitors true 2024-09-10T21:28:11.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:11.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:11.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:11.301 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:11.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:11.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:11.657 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 178 up_thru 178 down_at 175 last_clean_interval [166,174) [v2:127.0.0.1:6810/3921088946,v1:127.0.0.1:6811/3921088946] [v2:127.0.0.1:6812/3921088946,v1:127.0.0.1:6813/3921088946] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:11.658 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:11.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:11.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:11.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:11.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:11.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:11.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:12.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:12.028 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:12.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:12.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:12.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:28:12.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374917 2024-09-10T21:28:12.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374917 2024-09-10T21:28:12.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374917' 2024-09-10T21:28:12.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:12.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:28:12.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=764504178690 2024-09-10T21:28:12.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 764504178690 2024-09-10T21:28:12.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374917 1-764504178690' 2024-09-10T21:28:12.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:12.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374917 2024-09-10T21:28:12.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:12.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:28:12.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374917 2024-09-10T21:28:12.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:12.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374917 2024-09-10T21:28:12.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374917' 2024-09-10T21:28:12.264 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 738734374917 2024-09-10T21:28:12.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374916 -lt 738734374917 2024-09-10T21:28:12.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:28:13.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:28:13.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374917 -lt 738734374917 2024-09-10T21:28:13.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-764504178690 2024-09-10T21:28:13.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:13.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:28:13.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-764504178690 2024-09-10T21:28:13.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:13.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=764504178690 2024-09-10T21:28:13.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 764504178690' 2024-09-10T21:28:13.829 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 764504178690 2024-09-10T21:28:13.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:28:14.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 764504178690 -lt 764504178690 2024-09-10T21:28:14.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:28:14.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:14.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:28:14.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:28:14.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:28:14.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:28:14.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:28:14.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:28:14.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:28:14.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:28:14.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:28:14.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:28:14.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:14.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:15.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:28:15.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:28:15.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:28:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:28:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-10T21:28:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:28:15.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:28:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:28:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:28:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:15.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-09-10T21:28:16.412 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:28:16.412 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:28:16.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:28:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:28:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:28:16.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:28:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:28:16.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:28:16.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:28:16.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:28:16.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:28:16.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:28:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:28:16.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:28:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:28:16.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:16.956+0000 7f182e9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:16.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:16.958+0000 7f182e9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:16.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:16.959+0000 7f182e9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:28:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:28:17.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:17.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:17.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:18.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:18.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:18.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:28:18.491 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:28:18.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:18.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:18.586 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:18.584+0000 7f182e9b2640 -1 Falling back to public interface 2024-09-10T21:28:18.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:19.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:20.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:20.435 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:20.433+0000 7f182e9b2640 -1 osd.1 180 log_to_monitors true 2024-09-10T21:28:21.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:21.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:21.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:21.078 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:21.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:21.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 184 up_thru 184 down_at 181 last_clean_interval [178,180) [v2:127.0.0.1:6810/769528121,v1:127.0.0.1:6811/769528121] [v2:127.0.0.1:6812/769528121,v1:127.0.0.1:6813/769528121] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:21.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:21.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:21.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:21.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:21.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:21.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:21.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:21.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:21.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:21.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:21.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:21.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:21.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:21.871 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:21.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:21.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:21.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:28:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374920 2024-09-10T21:28:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374920 2024-09-10T21:28:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374920' 2024-09-10T21:28:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:22.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:28:22.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=790273982466 2024-09-10T21:28:22.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 790273982466 2024-09-10T21:28:22.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374920 1-790273982466' 2024-09-10T21:28:22.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:22.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374920 2024-09-10T21:28:22.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:22.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:28:22.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374920 2024-09-10T21:28:22.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:22.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374920 2024-09-10T21:28:22.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374920' 2024-09-10T21:28:22.174 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 738734374920 2024-09-10T21:28:22.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374919 -lt 738734374920 2024-09-10T21:28:22.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:28:23.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:28:23.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:23.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374920 -lt 738734374920 2024-09-10T21:28:23.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:23.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-790273982466 2024-09-10T21:28:23.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:23.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:28:23.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-790273982466 2024-09-10T21:28:23.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=790273982466 2024-09-10T21:28:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 790273982466' 2024-09-10T21:28:23.737 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 790273982466 2024-09-10T21:28:23.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:28:24.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 790273982466 -lt 790273982466 2024-09-10T21:28:24.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:28:24.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:24.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:24.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:28:24.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:28:24.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:28:24.372 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:28:24.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:28:24.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:28:24.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:28:24.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:28:24.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:28:24.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:28:24.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:24.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:24.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:28:24.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-10T21:28:24.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-10T21:28:24.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-10T21:28:24.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-10T21:28:24.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-10T21:28:24.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-10T21:28:24.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:28:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-10T21:28:25.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-09-10T21:28:25.386 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: {} 2024-09-10T21:28:25.386 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-10T21:28:25.402 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: {} 2024-09-10T21:28:25.402 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-10T21:28:25.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T21:28:25.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:28:25.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-10T21:28:25.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:28:25.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-10T21:28:25.414 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-10T21:28:25.414 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:28:25.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:28:25.708 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:28:25.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:28:25.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-10T21:28:25.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:25.711 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:28:25.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:28:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:28:25.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:28:25.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-10T21:28:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-10T21:28:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:28:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:28:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-10T21:28:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-10T21:28:26.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:28:26.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:28:26.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-10T21:28:26.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:28:26.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:28:26.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:28:26.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:28:26.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:28:26.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-10T21:28:26.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:28:26.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-10T21:28:26.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-10T21:28:26.365 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-10T21:28:26.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:28:26.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:28:26.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-10T21:28:26.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:28:26.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-10T21:28:26.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:28:26.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:26.646 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:28:26.647 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:28:26.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:28:26.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:28:26.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:28:26.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-10T21:28:26.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:28:26.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:28:26.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:28:26.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-10T21:28:26.821 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:28:27.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:28:27.126 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:28:27.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:28:27.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:28:27.129 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:28:27.130 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:27.130 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:27.130 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:27.130 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:28:27.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:28:27.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:28:27.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:28:27.215 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:27.216 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:27.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:28:27.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:28:27.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-10T21:28:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-09-10T21:28:27.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-10T21:28:27.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-10T21:28:27.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:28:27.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:28:27.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:28:27.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:28:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-10T21:28:27.833 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-10T21:28:27.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:27.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:28:27.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:27.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:28:27.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:28:27.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:28:27.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:28:27.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:28:27.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:28:27.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:28:27.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:28:28.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:28.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:28:29.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:28:29.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:28:29.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:28:29.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:28:29.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:28:29.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:28:29.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:28:29.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:29.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:28:30.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:28:30.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:28:30.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:28:30.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:28:30.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:28:30.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:28:30.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:28:30.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:30.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:28:31.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:28:31.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:28:31.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:28:31.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:28:31.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:28:31.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:28:31.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:28:28.436989+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-10T21:28:31.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-10T21:28:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-10T21:28:31.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-09-10T21:28:31.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-09-10T21:28:31.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-09-10T21:28:31.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:31.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:28:32.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-09-10T21:28:32.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-10T21:28:32.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:28:32.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-10T21:28:32.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:28:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-10T21:28:32.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-10T21:28:32.257 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:28:32.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=184 2024-09-10T21:28:32.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-10T21:28:32.260 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-10T21:28:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:28:32.284 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:28:32.285 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:28:32.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-10T21:28:32.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:28:32.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:28:32.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:28:32.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-09-10T21:28:32.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-10T21:28:32.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-10T21:28:32.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-09-10T21:28:32.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:28:32.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:28:32.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:28:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:28:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:28:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:28:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:28:32.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:28:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:28:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:28:32.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-10T21:28:34.228 INFO:tasks.workunit.client.0.smithi181.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-10T21:28:34.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:28:34.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:28:34.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:28:34.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:28:34.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:28:34.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:28:34.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:28:34.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:28:34.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:28:34.781 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:34.779+0000 7fde2b433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:34.783 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:34.781+0000 7fde2b433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:34.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:34.783+0000 7fde2b433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:35.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:28:35.042 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:28:35.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:35.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:35.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:36.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:36.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:36.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:28:36.353 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:28:36.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:36.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:36.418 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:36.416+0000 7fde2b433640 -1 Falling back to public interface 2024-09-10T21:28:36.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:37.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:28:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:28:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:37.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:38.042 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:38.040+0000 7fde2b433640 -1 osd.0 186 log_to_monitors true 2024-09-10T21:28:38.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:38.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 190 up_thru 190 down_at 187 last_clean_interval [172,186) [v2:127.0.0.1:6802/453220521,v1:127.0.0.1:6803/453220521] [v2:127.0.0.1:6804/453220521,v1:127.0.0.1:6805/453220521] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:39.339 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:39.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:39.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:39.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:39.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:39.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:39.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:39.757 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:39.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:39.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:39.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:28:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=816043786242 2024-09-10T21:28:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 816043786242 2024-09-10T21:28:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786242' 2024-09-10T21:28:39.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:39.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:28:40.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=790273982470 2024-09-10T21:28:40.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 790273982470 2024-09-10T21:28:40.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786242 1-790273982470' 2024-09-10T21:28:40.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:40.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-816043786242 2024-09-10T21:28:40.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:40.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:28:40.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-816043786242 2024-09-10T21:28:40.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:40.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=816043786242 2024-09-10T21:28:40.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 816043786242' 2024-09-10T21:28:40.011 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 816043786242 2024-09-10T21:28:40.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:40.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 816043786242 2024-09-10T21:28:40.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:28:41.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:28:41.286 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:41.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786242 -lt 816043786242 2024-09-10T21:28:41.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:41.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-790273982470 2024-09-10T21:28:41.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:41.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:28:41.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-790273982470 2024-09-10T21:28:41.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:41.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=790273982470 2024-09-10T21:28:41.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 790273982470' 2024-09-10T21:28:41.565 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 790273982470 2024-09-10T21:28:41.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:28:41.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 790273982471 -lt 790273982470 2024-09-10T21:28:41.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:28:41.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:41.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:28:42.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:28:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:28:42.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:28:42.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:42.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:42.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:28:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:28:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:28:42.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:28:42.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:28:42.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:28:42.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:28:42.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:28:42.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:28:42.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:43.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-09-10T21:28:44.150 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:28:44.150 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:28:44.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:28:44.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:28:44.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:28:44.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:28:44.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:28:44.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:28:44.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:28:44.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:28:44.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:28:44.674 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:28:44.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:28:44.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:44.699+0000 7f4f281c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:44.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:44.702+0000 7f4f281c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:44.705 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:44.703+0000 7f4f281c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:44.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:45.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:46.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:46.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:46.544+0000 7f4f281c5640 -1 Falling back to public interface 2024-09-10T21:28:46.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:47.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:47.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:48.032 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:48.030+0000 7f4f281c5640 -1 osd.1 192 log_to_monitors true 2024-09-10T21:28:48.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:48.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:48.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:48.843 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:48.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:48.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:28:49.135 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 196 up_thru 196 down_at 193 last_clean_interval [184,192) [v2:127.0.0.1:6810/747424406,v1:127.0.0.1:6811/747424406] [v2:127.0.0.1:6812/747424406,v1:127.0.0.1:6813/747424406] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:49.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:49.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:49.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:49.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:49.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:49.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:49.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:49.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:49.557 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:49.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:49.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:28:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=816043786245 2024-09-10T21:28:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 816043786245 2024-09-10T21:28:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786245' 2024-09-10T21:28:49.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:49.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:28:49.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=841813590018 2024-09-10T21:28:49.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 841813590018 2024-09-10T21:28:49.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786245 1-841813590018' 2024-09-10T21:28:49.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:49.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-816043786245 2024-09-10T21:28:49.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:49.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:28:49.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-816043786245 2024-09-10T21:28:49.766 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:49.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=816043786245 2024-09-10T21:28:49.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 816043786245' 2024-09-10T21:28:49.767 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 816043786245 2024-09-10T21:28:49.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:50.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786244 -lt 816043786245 2024-09-10T21:28:50.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:28:51.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:28:51.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:28:51.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786245 -lt 816043786245 2024-09-10T21:28:51.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:28:51.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-841813590018 2024-09-10T21:28:51.330 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:28:51.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:28:51.331 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-841813590018 2024-09-10T21:28:51.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:28:51.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=841813590018 2024-09-10T21:28:51.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 841813590018' 2024-09-10T21:28:51.333 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 841813590018 2024-09-10T21:28:51.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:28:51.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 841813590018 -lt 841813590018 2024-09-10T21:28:51.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:28:51.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:51.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:51.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:28:51.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:28:51.956 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:28:51.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:28:51.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:28:51.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:28:51.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:28:51.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:28:52.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:28:52.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:28:52.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:28:52.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:28:52.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:28:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:28:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:28:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-09-10T21:28:52.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:28:52.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:28:52.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:28:52.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:28:52.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:28:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:28:52.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:28:54.403 INFO:tasks.workunit.client.0.smithi181.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:28:54.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:28:54.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:28:54.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:28:54.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:28:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:28:54.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:28:54.958 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:54.956+0000 7fa32d4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:54.961 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:54.959+0000 7fa32d4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:54.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:54.960+0000 7fa32d4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:28:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:28:55.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:55.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:55.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:56.089 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:56.087+0000 7fa32d4f1640 -1 Falling back to public interface 2024-09-10T21:28:56.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:28:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:28:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:56.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:56.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:57.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:57.927 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:28:57.924+0000 7fa32d4f1640 -1 osd.0 198 log_to_monitors true 2024-09-10T21:28:58.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:28:59.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:28:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:28:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:28:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:28:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:28:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:28:59.547 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 202 up_thru 202 down_at 199 last_clean_interval [190,198) [v2:127.0.0.1:6802/4259385893,v1:127.0.0.1:6803/4259385893] [v2:127.0.0.1:6804/4259385893,v1:127.0.0.1:6805/4259385893] exists,up 788989a4-ba89-4ed0-84aa-663c6f11bae5 2024-09-10T21:28:59.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:28:59.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:28:59.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:28:59.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:28:59.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:28:59.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:28:59.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:28:59.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:28:59.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:28:59.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:28:59.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:28:59.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:28:59.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:28:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:28:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:28:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:28:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:28:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:28:59.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:28:59.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:28:59.949 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:28:59.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:28:59.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:28:59.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:29:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393794 2024-09-10T21:29:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393794 2024-09-10T21:29:00.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393794' 2024-09-10T21:29:00.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:29:00.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:29:00.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=841813590021 2024-09-10T21:29:00.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 841813590021 2024-09-10T21:29:00.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393794 1-841813590021' 2024-09-10T21:29:00.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:00.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393794 2024-09-10T21:29:00.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:00.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:29:00.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393794 2024-09-10T21:29:00.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393794 2024-09-10T21:29:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393794' 2024-09-10T21:29:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 867583393794 2024-09-10T21:29:00.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:29:00.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 867583393794 2024-09-10T21:29:00.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:29:01.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:29:01.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:29:01.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393794 -lt 867583393794 2024-09-10T21:29:01.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:01.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-841813590021 2024-09-10T21:29:01.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:01.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:29:01.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-841813590021 2024-09-10T21:29:01.725 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:01.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=841813590021 2024-09-10T21:29:01.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 841813590021' 2024-09-10T21:29:01.726 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 841813590021 2024-09-10T21:29:01.726 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:29:02.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 841813590021 -lt 841813590021 2024-09-10T21:29:02.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:29:02.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:02.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:02.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:29:02.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:29:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:29:02.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:29:02.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:29:02.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:02.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:29:03.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:29:03.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:29:03.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-10T21:29:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-10T21:29:04.581 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:29:05.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:05.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:29:05.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:29:05.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:29:05.102 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:29:05.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:29:05.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:29:05.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:29:05.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:29:05.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:29:05.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:29:05.130 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:05.127+0000 7f8d79618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:05.132 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:05.130+0000 7f8d79618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:05.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:05.131+0000 7f8d79618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:05.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:29:05.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:05.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:29:05.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:06.503 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:06.500+0000 7f8d79618640 -1 Falling back to public interface 2024-09-10T21:29:06.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:06.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:06.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:29:06.659 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:29:06.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:06.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:29:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:07.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:07.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:07.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:29:07.944 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:29:07.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:07.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:29:08.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:08.458 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:08.456+0000 7f8d79618640 -1 osd.1 204 log_to_monitors true 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:29:09.579 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 208 up_thru 208 down_at 205 last_clean_interval [196,204) [v2:127.0.0.1:6810/3960102752,v1:127.0.0.1:6811/3960102752] [v2:127.0.0.1:6812/3960102752,v1:127.0.0.1:6813/3960102752] exists,up 6765db78-c000-4cda-9031-a1dd5c80194a 2024-09-10T21:29:09.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:29:09.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:29:09.581 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:29:09.581 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:29:09.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:29:09.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:29:09.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:29:09.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:29:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:29:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:29:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:29:09.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:29:09.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:29:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:29:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:29:09.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:29:09.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:29:09.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:29:10.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393797 2024-09-10T21:29:10.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393797 2024-09-10T21:29:10.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393797' 2024-09-10T21:29:10.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:29:10.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:29:10.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=893353197570 2024-09-10T21:29:10.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 893353197570 2024-09-10T21:29:10.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393797 1-893353197570' 2024-09-10T21:29:10.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:10.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393797 2024-09-10T21:29:10.179 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:10.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:29:10.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393797 2024-09-10T21:29:10.180 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:10.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393797 2024-09-10T21:29:10.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393797' 2024-09-10T21:29:10.181 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 867583393797 2024-09-10T21:29:10.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:29:10.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393796 -lt 867583393797 2024-09-10T21:29:10.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:29:11.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:29:11.455 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:29:11.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393797 -lt 867583393797 2024-09-10T21:29:11.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:11.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-893353197570 2024-09-10T21:29:11.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:11.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:29:11.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-893353197570 2024-09-10T21:29:11.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:11.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=893353197570 2024-09-10T21:29:11.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 893353197570' 2024-09-10T21:29:11.742 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 893353197570 2024-09-10T21:29:11.742 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:29:12.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 893353197570 -lt 893353197570 2024-09-10T21:29:12.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:29:12.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:12.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:12.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:29:12.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:29:12.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:29:12.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:29:12.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:29:12.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:29:12.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:29:12.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:29:12.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:29:12.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:29:12.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:12.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:13.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:29:13.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:29:13.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:29:13.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-09-10T21:29:13.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-10T21:29:13.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:29:13.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-10T21:29:13.006 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-10T21:29:13.006 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:29:13.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:29:13.299 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:29:13.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:29:13.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:29:13.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:29:13.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T21:29:13.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:29:13.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:29:13.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-10T21:29:13.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:29:13.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:13.302 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:29:13.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:29:13.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:29:13.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:29:13.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:29:13.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-10T21:29:13.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-10T21:29:13.475 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:29:13.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-10T21:29:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-10T21:29:13.476 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:29:13.750 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:29:13.751 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:29:13.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:29:13.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:29:13.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:29:13.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:29:13.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:29:13.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-10T21:29:13.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:29:13.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:29:13.839 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:13.840 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:13.840 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:13.840 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:29:13.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:29:13.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-10T21:29:13.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:29:13.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-10T21:29:13.929 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:29:14.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:29:14.240 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:29:14.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:29:14.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:29:14.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:29:14.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-10T21:29:14.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:29:14.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:29:14.241 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:14.243 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:14.244 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:14.244 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:29:14.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:29:14.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:29:14.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:29:14.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.0.asok 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:29:14.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T21:29:14.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-10T21:29:14.419 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:29:14.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:29:14.696 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T21:29:14.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-10T21:29:14.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T21:29:14.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T21:29:14.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T21:29:14.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:29:14.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52449/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T21:29:14.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T21:29:14.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:14.786 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:14.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-osd.1.asok 2024-09-10T21:29:14.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T21:29:14.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52449/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-10T21:29:14.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-09-10T21:29:14.967 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: {} 2024-09-10T21:29:14.968 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0: osd_max_object_size = '' 2024-09-10T21:29:14.984 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: {} 2024-09-10T21:29:14.984 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1: osd_max_object_size = '' 2024-09-10T21:29:14.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-10T21:29:14.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-10T21:29:14.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:29:14.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:14.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:29:14.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:14.994 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:29:15.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:15.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-10T21:29:15.526 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-10T21:24:35.481450+0000 last_deep_scrub_stamp 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:29:15.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:15.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:29:15.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:15.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:29:15.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:15.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:29:15.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:15.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:16.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:16.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:16.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:29:16.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:16.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:29:16.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:16.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:29:17.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:17.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:18.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:18.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:18.095 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:29:18.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:18.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:29:18.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:18.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:29:18.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:24:35.481450+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:18.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:19.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:19.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:19.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-10T21:29:19.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:19.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-10T21:29:19.380 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:19.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-10T21:29:19.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:16.585003+0000 '>' 2024-09-10T21:24:35.481450+0000 2024-09-10T21:29:19.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:29:19.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-10T21:29:19.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-10T21:29:19.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-09-10T21:29:19.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-09-10T21:29:19.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:19.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:19.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:20.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:20.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:20.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:20.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:20.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-10T21:29:20.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-09-10T21:29:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-10T21:29:20.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:29:20.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-10T21:29:20.078 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:29:20.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-10T21:29:20.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-10T21:29:20.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-10T21:29:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=208 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:29:20.112 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-10T21:29:20.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:29:20.136 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-10T21:29:20.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:29:20.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:29:20.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:29:20.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-09-10T21:29:20.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-10T21:29:20.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-09-10T21:29:20.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=3.0 2024-09-10T21:29:20.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 3.0 2024-09-10T21:29:20.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:20.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:20.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:20.162 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:20.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 3.0 2024-09-10T21:29:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0 on osd.1 to repair 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 3.0 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:20.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:20.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:20.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:16.585003+0000 '>' 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:20.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:21.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:21.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:21.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:29:21.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:21.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:21.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:21.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:16.585003+0000 '>' 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:22.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:23.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:23.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:23.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:29:23.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:23.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:23.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:23.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:23.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:16.585003+0000 '>' 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:23.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:24.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:24.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:24.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:29:24.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:24.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:24.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:24.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:24.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:16.585003+0000 '>' 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:24.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:29:25.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:29:25.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:29:25.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:29:25.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:29:25.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:29:25.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:29:25.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:29:26.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:29:22.445445+0000 '>' 2024-09-10T21:29:16.585003+0000 2024-09-10T21:29:26.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:29:26.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:29:28.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-10T21:29:28.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:29:28.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:29:28.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:29:28.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:29:28.145 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:29:28.145 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:29:28.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:29:28.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:29:28.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:29:28.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:29:28.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:29:28.527 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:29:28.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:29:28.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:29:28.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:29:28.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393802 2024-09-10T21:29:28.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393802 2024-09-10T21:29:28.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393802' 2024-09-10T21:29:28.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:29:28.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:29:28.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=893353197575 2024-09-10T21:29:28.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 893353197575 2024-09-10T21:29:28.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393802 1-893353197575' 2024-09-10T21:29:28.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:28.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393802 2024-09-10T21:29:28.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:28.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:29:28.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393802 2024-09-10T21:29:28.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:28.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393802 2024-09-10T21:29:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 867583393802 2024-09-10T21:29:28.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393802' 2024-09-10T21:29:28.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:29:29.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393802 -lt 867583393802 2024-09-10T21:29:29.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:29:29.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-893353197575 2024-09-10T21:29:29.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:29:29.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:29:29.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-893353197575 2024-09-10T21:29:29.016 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:29:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=893353197575 2024-09-10T21:29:29.018 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 893353197575 2024-09-10T21:29:29.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 893353197575' 2024-09-10T21:29:29.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:29:29.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 893353197575 -lt 893353197575 2024-09-10T21:29:29.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:29:29.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:29.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:29.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:29:29.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:29:29.649 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:29:29.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:29:29.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:29:29.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:29:29.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:29:30.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:29:30.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:29:30.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:29:30.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-10T21:29:30.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-10T21:29:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-10T21:29:30.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-10T21:29:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-10T21:29:30.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-10T21:29:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-09-10T21:29:30.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-10T21:29:30.708 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'csr_pool' removed 2024-09-10T21:29:30.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:29:30.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:29:30.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:29:30.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:29:30.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:29:30.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:29:30.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:29:30.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:29:30.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:29:30.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:29:30.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:29:30.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:29:30.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:29:30.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:29:30.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:29:30.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:29:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:29:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:29:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:29:30.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:29:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:29:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:29:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:29:30.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:29:30.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:30.870 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:30.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:29:30.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:29:30.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:29:30.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:29:30.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:29:30.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:29:30.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:29:30.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:29:30.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:29:30.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:29:30.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:29:30.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:29:30.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:29:30.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:29:30.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:29:30.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:29:30.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:29:30.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:29:30.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:29:30.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:29:30.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:30.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:30.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:29:30.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:29:30.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:29:30.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:29:30.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:29:30.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:30.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:30.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-09-10T21:29:30.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5934: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:29:30.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-10T21:29:31.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:29:31.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:31.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:29:31.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:29:31.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:29:31.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:29:31.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:29:31.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:29:31.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:29:31.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:29:31.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:31.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:29:31.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:29:31.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:29:31.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:29:32.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:32.495 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:29:32.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:29:32.526 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:29:32.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:29:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:29:32.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:29:32.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e992e7ea-88e1-4c76-bd7a-6f45e9479e1d 2024-09-10T21:29:32.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e992e7ea-88e1-4c76-bd7a-6f45e9479e1d' 2024-09-10T21:29:32.530 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 e992e7ea-88e1-4c76-bd7a-6f45e9479e1d 2024-09-10T21:29:32.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:29:32.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8uuBmKIB0IBAAd1oUXB+g3RrgE91fQ/snHQ== 2024-09-10T21:29:32.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8uuBmKIB0IBAAd1oUXB+g3RrgE91fQ/snHQ=="}' 2024-09-10T21:29:32.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e992e7ea-88e1-4c76-bd7a-6f45e9479e1d -i td/osd-scrub-repair/0/new.json 2024-09-10T21:29:32.750 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:29:32.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:29:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8uuBmKIB0IBAAd1oUXB+g3RrgE91fQ/snHQ== --osd-uuid e992e7ea-88e1-4c76-bd7a-6f45e9479e1d 2024-09-10T21:29:32.790 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:32.788+0000 7f0ad5414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:32.801 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:32.799+0000 7f0ad5414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:32.800+0000 7f0ad5414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:32.802 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:32.801+0000 7f0ad5414640 -1 bdev(0x55c20a125c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:29:32.803 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:32.801+0000 7f0ad5414640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:29:38.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:29:38.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:29:38.275 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:29:38.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:29:38.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:29:38.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:29:38.636 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:29:38.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:29:38.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:29:38.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:29:38.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:29:38.664 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:38.662+0000 7ff39473e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:38.683 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:38.681+0000 7ff39473e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:38.684 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:38.682+0000 7ff39473e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:29:38.913 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:29:38.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:38.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:39.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:40.043 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:40.041+0000 7ff39473e640 -1 Falling back to public interface 2024-09-10T21:29:40.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:40.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:40.191 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:29:40.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:29:40.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:40.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:40.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:41.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:41.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:41.484 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:29:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:29:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:41.633 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:29:41.631+0000 7ff39473e640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:29:41.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:42.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:42.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:42.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:29:42.809 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:29:42.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:42.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:43.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:44.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:44.099 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:29:44.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:29:44.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:44.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:44.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:45.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:45.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:45.391 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:29:45.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:29:45.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:45.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:45.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:46.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:46.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:46.680 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:29:46.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:29:46.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:46.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:46.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:47.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:47.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:47.968 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:29:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:29:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:48.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:49.258 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:29:49.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:29:49.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:49.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:49.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:50.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:50.619 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:29:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:29:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:50.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:50.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:51.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:51.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:51.907 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:29:51.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:29:51.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:51.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:52.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:53.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:53.207 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:29:53.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:29:53.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:53.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:53.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:29:54.886 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:29:54.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:54.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:55.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:56.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:56.597 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:29:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:29:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:56.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:57.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:29:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:29:58.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:29:58.441 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:29:58.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:29:58.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:29:58.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:29:59.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:00.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:00.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:00.261 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:30:00.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:30:00.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:00.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:30:00.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:01.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:01.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:01.628 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:30:01.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:30:01.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:01.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:30:02.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:03.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:03.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:03.423 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:30:03.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:30:03.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:03.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:30:03.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:04.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:30:04.995 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2300009153,v1:127.0.0.1:6803/2300009153] [v2:127.0.0.1:6804/2300009153,v1:127.0.0.1:6805/2300009153] exists,up e992e7ea-88e1-4c76-bd7a-6f45e9479e1d 2024-09-10T21:30:04.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:30:04.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:30:04.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:30:04.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:30:05.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:30:05.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:30:05.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6d0e96f0-b603-4a3a-be85-fdf23e8782ba 2024-09-10T21:30:05.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6d0e96f0-b603-4a3a-be85-fdf23e8782ba' 2024-09-10T21:30:05.002 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 6d0e96f0-b603-4a3a-be85-fdf23e8782ba 2024-09-10T21:30:05.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:30:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBduuBmagUKARAAX08pQkJrw/a2gJCIic+pBw== 2024-09-10T21:30:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBduuBmagUKARAAX08pQkJrw/a2gJCIic+pBw=="}' 2024-09-10T21:30:05.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6d0e96f0-b603-4a3a-be85-fdf23e8782ba -i td/osd-scrub-repair/1/new.json 2024-09-10T21:30:05.344 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:30:05.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:30:05.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBduuBmagUKARAAX08pQkJrw/a2gJCIic+pBw== --osd-uuid 6d0e96f0-b603-4a3a-be85-fdf23e8782ba 2024-09-10T21:30:05.386 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:05.384+0000 7f13df7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:05.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:05.440+0000 7f13df7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:05.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:05.441+0000 7f13df7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:05.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:05.441+0000 7f13df7c5640 -1 bdev(0x557e71f39c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:30:05.443 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:05.441+0000 7f13df7c5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:30:11.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:30:11.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:30:11.856 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:30:11.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:30:11.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:30:12.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:30:12.214 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:30:12.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:30:12.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:30:12.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:30:12.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:30:12.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:12.240+0000 7fd12ca33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:12.244 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:12.242+0000 7fd12ca33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:12.245 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:12.244+0000 7fd12ca33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:12.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:30:12.520 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:30:12.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:12.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:12.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:13.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:13.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:13.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:30:13.796 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:30:13.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:13.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:13.849 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:13.848+0000 7fd12ca33640 -1 Falling back to public interface 2024-09-10T21:30:14.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:15.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:15.082 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:30:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:30:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:15.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:15.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:15.474 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:15.472+0000 7fd12ca33640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:30:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:16.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:16.365 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:30:16.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:30:16.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:16.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:16.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:17.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:17.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:17.797 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:30:17.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:30:17.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:17.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:18.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:19.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:19.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:19.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:30:19.133 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:30:19.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:19.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:20.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:20.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:20.420 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:30:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:30:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:20.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:20.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:21.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:21.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:21.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:30:21.732 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:30:21.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:21.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:22.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:23.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:23.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:23.018 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:30:23.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:30:23.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:23.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:23.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:24.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:24.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:24.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:30:24.313 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:30:24.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:24.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:24.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:25.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:25.601 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:30:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:30:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:25.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:25.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:26.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:26.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:26.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:30:26.891 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:30:26.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:26.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:27.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:28.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:28.302 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:30:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:30:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:28.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:29.930 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:30:29.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:30:29.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:29.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:30.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:31.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:31.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:31.562 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:30:31.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:30:31.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:31.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:32.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:33.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:33.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:33.270 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:30:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:30:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:33.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:33.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:34.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:34.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:34.610 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:30:34.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:30:34.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:34.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:35.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:36.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:36.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:36.371 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:30:36.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:30:36.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:36.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:36.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:37.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:37.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:37.781 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:30:37.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:30:37.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:37.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:38.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:30:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:30:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:39.479 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:30:39.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:30:39.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:39.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:39.773 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3159061532,v1:127.0.0.1:6811/3159061532] [v2:127.0.0.1:6812/3159061532,v1:127.0.0.1:6813/3159061532] exists,up 6d0e96f0-b603-4a3a-be85-fdf23e8782ba 2024-09-10T21:30:39.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:30:39.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:30:39.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:30:39.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-09-10T21:30:39.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:30:40.040 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:30:40.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:30:40.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:30:40.440 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:30:40.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:30:41.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:30:43.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-10T21:30:43.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:30:43.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:30:43.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:30:43.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:30:43.168 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:30:43.169 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:30:43.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:30:43.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:30:43.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:30:43.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:30:43.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:30:43.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:30:43.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:30:43.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:30:43.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:30:43.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:30:43.552 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:30:43.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:30:43.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:30:43.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:30:43.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-10T21:30:43.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-10T21:30:43.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-10T21:30:43.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:30:43.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:30:43.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672964 2024-09-10T21:30:43.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672964 2024-09-10T21:30:43.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494 1-42949672964' 2024-09-10T21:30:43.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:30:43.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-10T21:30:43.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:30:43.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:30:43.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-10T21:30:43.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:30:43.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-10T21:30:43.762 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836494 2024-09-10T21:30:43.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-10T21:30:43.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:30:44.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-10T21:30:44.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:30:45.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:30:45.040 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:30:45.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-10T21:30:45.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:30:45.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672964 2024-09-10T21:30:45.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:30:45.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:30:45.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672964 2024-09-10T21:30:45.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:30:45.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672964 2024-09-10T21:30:45.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-10T21:30:45.328 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672964 2024-09-10T21:30:45.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:30:45.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-10T21:30:45.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:30:45.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:30:45.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:30:45.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:30:45.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:30:45.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:30:45.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:30:45.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:30:45.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:30:45.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:30:45.980 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:30:46.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:30:46.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:30:46.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:30:46.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:30:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:30:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:30:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:30:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5945: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-09-10T21:30:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-10T21:30:46.976 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'foo' created 2024-09-10T21:30:46.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:30:47.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-09-10T21:30:47.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-10T21:30:48.379 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'csr_pool' created 2024-09-10T21:30:48.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:30:49.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-10T21:30:49.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:30:49.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:30:49.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:30:49.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:30:49.498 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:30:49.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:30:49.807 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:30:49.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:30:49.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:30:49.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:30:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-10T21:30:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-10T21:30:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-10T21:30:49.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:30:49.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:30:50.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-10T21:30:50.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-10T21:30:50.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-42949672967' 2024-09-10T21:30:50.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:30:50.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-10T21:30:50.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:30:50.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:30:50.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-10T21:30:50.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:30:50.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-10T21:30:50.051 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836497 2024-09-10T21:30:50.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-10T21:30:50.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:30:50.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836497 2024-09-10T21:30:50.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:30:51.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:30:51.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:30:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-10T21:30:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:30:51.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-10T21:30:51.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:30:51.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:30:51.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-10T21:30:51.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:30:51.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-10T21:30:51.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-10T21:30:51.617 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672967 2024-09-10T21:30:51.617 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:30:51.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-10T21:30:51.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:30:51.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:30:51.895 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:30:52.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:30:52.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:30:52.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:30:52.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:30:52.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:30:52.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:30:52.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:30:52.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:30:52.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:30:52.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:30:52.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:30:52.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:30:53.252 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:30:53.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:30:53.713 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:30:53.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:30:53.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:30:53.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:30:53.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-10T21:30:54.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-10T21:30:54.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-10T21:30:54.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-10T21:30:54.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-10T21:30:54.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:30:54.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-10T21:30:54.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-10T21:30:54.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:30:54.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:30:54.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:30:54.532 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:30:54.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:30:54.942 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:30:54.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:30:54.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:30:54.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:30:55.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-10T21:30:55.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-10T21:30:55.498 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-09-10T21:30:55.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-10T21:30:55.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-10T21:30:55.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-10T21:30:55.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:30:55.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-09-10T21:30:55.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-09-10T21:30:55.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-10T21:30:55.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-10T21:30:55.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-10T21:30:55.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:30:56.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-09-10T21:30:56.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-09-10T21:30:56.306 INFO:tasks.workunit.client.0.smithi181.stdout:created pool csr_pool snap snap1 2024-09-10T21:30:56.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-09-10T21:30:56.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-10T21:30:56.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-10T21:30:56.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-10T21:30:56.310 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-09-10T21:30:56.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-09-10T21:30:56.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-10T21:30:56.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5971: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-09-10T21:30:56.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:30:56.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:30:56.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:30:56.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:30:56.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:30:56.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:30:56.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:30:56.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-09-10T21:30:58.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:30:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:30:58.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:30:58.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:30:58.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:30:58.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:30:58.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:58.935+0000 7f4ffd5d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:58.939 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:58.938+0000 7f4ffd5d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:58.941 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:30:58.939+0000 7f4ffd5d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:30:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:30:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:30:59.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:00.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:00.301+0000 7f4ffd5d4640 -1 Falling back to public interface 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:00.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:31:00.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:01.657 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:01.656+0000 7f4ffd5d4640 -1 osd.1 30 log_to_monitors true 2024-09-10T21:31:01.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:01.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:01.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:31:01.762 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:31:01.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:01.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:31:02.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:03.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:31:03.357 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/2145831566,v1:127.0.0.1:6811/2145831566] [v2:127.0.0.1:6812/2145831566,v1:127.0.0.1:6813/2145831566] exists,up 6d0e96f0-b603-4a3a-be85-fdf23e8782ba 2024-09-10T21:31:03.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:31:03.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:31:03.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:31:03.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:31:03.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:31:03.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:31:03.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:31:03.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:31:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:31:03.359 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:31:03.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:31:03.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:31:03.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:31:03.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:31:03.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:31:03.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:31:03.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:31:03.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:31:03.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:31:03.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:31:03.727 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:31:03.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:31:03.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:31:03.728 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:31:03.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-10T21:31:03.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-10T21:31:03.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-10T21:31:03.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:31:03.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:31:03.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-10T21:31:03.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-10T21:31:03.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-146028888066' 2024-09-10T21:31:03.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:31:03.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-10T21:31:03.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:31:03.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:31:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-10T21:31:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:31:03.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-10T21:31:03.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-10T21:31:03.936 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 21474836500 2024-09-10T21:31:03.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:31:04.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-10T21:31:04.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:31:05.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:31:05.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:31:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-10T21:31:05.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:31:05.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888066 2024-09-10T21:31:05.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:31:05.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:31:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888066 2024-09-10T21:31:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:31:05.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-10T21:31:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-09-10T21:31:05.504 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 146028888066 2024-09-10T21:31:05.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:31:05.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-10T21:31:05.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:31:05.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:31:05.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:31:06.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:31:06.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:31:06.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:31:06.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:31:06.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:31:06.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:31:06.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:31:06.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:31:06.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:31:06.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:31:06.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:31:06.403 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-10T21:31:06.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-09-10T21:31:06.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-09-10T21:31:06.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-10T21:31:06.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5977: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:31:06.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:31:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:31:07.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:31:07.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-10T21:31:07.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:31:07.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:31:07.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:31:07.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:31:07.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:31:07.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-09-10T21:31:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:31:09.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:31:09.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:31:09.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:31:09.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:31:09.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:31:09.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:31:09.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:31:09.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:31:09.271 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:09.269+0000 7f9a2ba0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:09.272+0000 7f9a2ba0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:09.273+0000 7f9a2ba0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:31:09.528 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:31:09.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:09.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:09.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:10.408 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:10.407+0000 7f9a2ba0e640 -1 Falling back to public interface 2024-09-10T21:31:10.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:10.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:10.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:31:10.808 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:31:10.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:10.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:11.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:11.728 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:11.726+0000 7f9a2ba0e640 -1 osd.0 36 log_to_monitors true 2024-09-10T21:31:12.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:12.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:12.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:31:12.103 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:31:12.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:12.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 40 up_thru 34 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/1428398017,v1:127.0.0.1:6803/1428398017] [v2:127.0.0.1:6804/1428398017,v1:127.0.0.1:6805/1428398017] exists,up e992e7ea-88e1-4c76-bd7a-6f45e9479e1d 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:31:12.386 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:31:12.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:31:12.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:31:12.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:31:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:31:12.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:31:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:31:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:31:12.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:31:12.482 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:31:12.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:31:12.761 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:31:12.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:31:12.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:31:12.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:31:12.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-10T21:31:12.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-10T21:31:12.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842' 2024-09-10T21:31:12.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:31:12.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888069 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888069 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-146028888069' 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691842 2024-09-10T21:31:13.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:31:13.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:31:13.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691842 2024-09-10T21:31:13.029 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:31:13.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-10T21:31:13.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-09-10T21:31:13.030 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 171798691842 2024-09-10T21:31:13.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:31:13.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 171798691842 2024-09-10T21:31:13.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:31:14.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:31:14.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:31:14.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-10T21:31:14.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:31:14.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888069 2024-09-10T21:31:14.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:31:14.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:31:14.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888069 2024-09-10T21:31:14.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:31:14.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888069 2024-09-10T21:31:14.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-09-10T21:31:14.625 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 146028888069 2024-09-10T21:31:14.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:31:14.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-10T21:31:14.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:31:14.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:31:14.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:31:15.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:31:15.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:31:15.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:31:15.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:31:15.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:31:15.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:31:15.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:31:15.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:31:15.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:31:15.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:31:15.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:31:15.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:31:15.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:31:15.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:31:15.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:31:15.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5982: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5984: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:31:15.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:31:16.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:16.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-10T21:31:16.414 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-10T21:31:16.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:16.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-10T21:31:16.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:16.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:31:16.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:31:16.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:31:16.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:31:16.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:31:16.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:31:16.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:31:16.428 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:31:16.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:30:48.320482+0000 '>' 2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:16.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:31:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:31:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:31:17.710 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:31:17.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:31:17.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:31:17.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:31:17.711 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:31:18.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:30:48.320482+0000 '>' 2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:18.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:31:19.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:31:19.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:31:19.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-10T21:31:19.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-10T21:31:19.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:31:19.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:31:19.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-10T21:31:19.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:31:16.755024+0000 '>' 2024-09-10T21:30:48.320482+0000 2024-09-10T21:31:19.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:31:19.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5986: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-09-10T21:31:19.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-09-10T21:31:19.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-09-10T21:31:19.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-10T21:31:19.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-09-10T21:31:19.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5992: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-09-10T21:31:19.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-10T21:31:19.372 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:31:19.372 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:31:19.372 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-10T21:31:19.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:31:19.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-10T21:31:19.395 INFO:tasks.workunit.client.0.smithi181.stderr: . as $in 2024-09-10T21:31:19.395 INFO:tasks.workunit.client.0.smithi181.stderr: | if type == "object" then 2024-09-10T21:31:19.395 INFO:tasks.workunit.client.0.smithi181.stderr: reduce keys[] as $key 2024-09-10T21:31:19.395 INFO:tasks.workunit.client.0.smithi181.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr: else f 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr: end; 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-10T21:31:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-10T21:31:19.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-10T21:31:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6161: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:31:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-10T21:31:19.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6162: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6167: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6177: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6179: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-10T21:31:19.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-09-10T21:31:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-09-10T21:31:19.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6196: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-10T21:31:19.891 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'csr_pool' removed 2024-09-10T21:31:19.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:31:19.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:31:19.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:31:19.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:31:19.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:31:19.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:31:19.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:31:19.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:31:19.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:31:20.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:31:20.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:31:20.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:31:20.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:31:20.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:31:20.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:31:20.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:31:20.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:31:20.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:31:20.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:31:20.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:31:20.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:31:20.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:31:20.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:31:20.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:31:20.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:31:20.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:31:20.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:31:20.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:31:20.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:31:20.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:31:20.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:31:20.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:31:20.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:31:20.037 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:31:20.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:31:20.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:31:20.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:31:20.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:31:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:31:20.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:31:20.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:31:20.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:31:20.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:31:20.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:31:20.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:31:20.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:31:20.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:31:20.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-09-10T21:31:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:31:20.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:31:20.499 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:20.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:31:20.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:31:20.693 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:31:20.694 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:31:20.694 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:31:20.694 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:31:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.695 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:31:20.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:31:20.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:31:20.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:31:20.774 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:31:20.774 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:20.774 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:20.774 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:31:20.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:31:20.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:31:20.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:31:21.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:31:21.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:31:21.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:31:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:31:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:21.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:21.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:31:21.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:31:21.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:31:21.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:21.706 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:31:21.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:31:21.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:31:21.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fcebd4d9-d257-46df-b375-663e05d013df 2024-09-10T21:31:21.708 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 fcebd4d9-d257-46df-b375-663e05d013df 2024-09-10T21:31:21.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fcebd4d9-d257-46df-b375-663e05d013df' 2024-09-10T21:31:21.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:31:21.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCpuuBm7PPwKhAASq0P9Gyjsy+bGAgixK/DbQ== 2024-09-10T21:31:21.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCpuuBm7PPwKhAASq0P9Gyjsy+bGAgixK/DbQ=="}' 2024-09-10T21:31:21.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fcebd4d9-d257-46df-b375-663e05d013df -i td/osd-scrub-repair/0/new.json 2024-09-10T21:31:21.924 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:31:21.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:31:21.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpuuBm7PPwKhAASq0P9Gyjsy+bGAgixK/DbQ== --osd-uuid fcebd4d9-d257-46df-b375-663e05d013df 2024-09-10T21:31:21.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:21.963+0000 7fb9d4f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:21.992 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:21.990+0000 7fb9d4f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:21.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:21.992+0000 7fb9d4f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:21.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:21.992+0000 7fb9d4f58640 -1 bdev(0x55c0b27bfc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:31:21.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:21.992+0000 7fb9d4f58640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:31:27.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:31:27.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:31:27.911 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:31:27.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:31:27.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:31:28.272 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:31:28.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:31:28.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:31:28.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:31:28.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:31:28.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:31:28.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:28.299+0000 7f7752017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:28.302 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:28.301+0000 7f7752017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:28.303 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:28.302+0000 7f7752017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:28.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:31:28.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:31:28.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:31:28.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:28.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:28.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:29.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:29.648+0000 7f7752017640 -1 Falling back to public interface 2024-09-10T21:31:29.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:29.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:29.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:31:29.824 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:31:29.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:29.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:30.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:31.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:31.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:31.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:31:31.113 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:31:31.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:31.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:31.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:31.820+0000 7f7752017640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:31:32.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:32.398 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:31:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:31:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:32.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:32.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:33.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:33.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:33.766 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:31:33.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:31:33.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:33.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:34.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:35.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:35.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:35.055 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:31:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:31:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:35.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:35.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:36.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:36.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:36.378 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:31:36.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:31:36.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:36.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:36.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:37.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:37.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:37.666 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:31:37.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:31:37.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:37.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:37.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:38.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:38.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:38.954 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:31:38.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:31:38.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:38.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:39.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:40.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:40.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:40.245 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:31:40.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:31:40.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:40.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:40.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:41.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:41.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:41.581 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:31:41.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:31:41.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:41.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:41.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:42.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:42.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:42.871 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:31:42.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:31:42.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:42.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:43.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:44.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:44.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:31:44.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:31:44.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:44.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:44.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:45.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:45.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:45.482 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:31:45.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:31:45.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:45.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:45.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:46.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:46.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:46.875 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:31:46.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:31:46.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:46.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:47.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:48.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:48.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:48.448 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:31:48.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:31:48.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:48.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:49.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:50.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:50.109 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:31:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:31:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:50.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:51.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:52.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:52.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:52.028 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:31:52.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:31:52.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:52.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:52.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:53.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:53.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:53.471 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:31:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:31:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:53.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:31:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:31:54.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:31:54.967 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:31:54.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:31:54.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:31:54.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:31:55.253 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2443619627,v1:127.0.0.1:6803/2443619627] [v2:127.0.0.1:6804/2443619627,v1:127.0.0.1:6805/2443619627] exists,up fcebd4d9-d257-46df-b375-663e05d013df 2024-09-10T21:31:55.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:31:55.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:31:55.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:31:55.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:31:55.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:31:55.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2240e0a1-1fe3-451a-a329-2493ac8f29e7 2024-09-10T21:31:55.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2240e0a1-1fe3-451a-a329-2493ac8f29e7' 2024-09-10T21:31:55.261 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 2240e0a1-1fe3-451a-a329-2493ac8f29e7 2024-09-10T21:31:55.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:31:55.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDLuuBmEANbEBAACnTOPxxPbJXOKpv6CwEBJA== 2024-09-10T21:31:55.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDLuuBmEANbEBAACnTOPxxPbJXOKpv6CwEBJA=="}' 2024-09-10T21:31:55.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2240e0a1-1fe3-451a-a329-2493ac8f29e7 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:31:55.600 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:31:55.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:31:55.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLuuBmEANbEBAACnTOPxxPbJXOKpv6CwEBJA== --osd-uuid 2240e0a1-1fe3-451a-a329-2493ac8f29e7 2024-09-10T21:31:55.641 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:55.640+0000 7f3620d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:55.676 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:55.674+0000 7f3620d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:55.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:55.675+0000 7f3620d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:31:55.677 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:55.676+0000 7f3620d89640 -1 bdev(0x5604abd59c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:31:55.678 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:31:55.676+0000 7f3620d89640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:32:01.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:32:01.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:32:01.638 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:32:01.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:32:02.024 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:32:02.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:32:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:32:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:32:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:32:02.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:32:02.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:02.049+0000 7f8eb9d1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:02.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:02.051+0000 7f8eb9d1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:02.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:02.053+0000 7f8eb9d1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:02.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:32:02.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:32:02.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:32:02.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:02.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:02.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:03.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:32:03.578 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:32:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:03.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:03.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:03.911 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:03.910+0000 7f8eb9d1d640 -1 Falling back to public interface 2024-09-10T21:32:04.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:04.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:04.871 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:32:04.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:32:04.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:04.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:05.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:05.848 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:05.846+0000 7f8eb9d1d640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:32:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:06.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:32:06.156 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:32:06.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:06.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:06.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:07.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:32:07.448 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:32:07.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:07.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:07.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:08.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:08.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:08.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:32:08.741 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:32:08.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:08.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:09.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:10.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:10.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:10.030 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:32:10.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:32:10.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:10.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:10.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:11.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:11.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:11.318 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:32:11.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:32:11.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:11.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:11.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:12.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:12.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:12.607 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:32:12.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:32:12.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:12.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:12.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:13.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:13.955 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:32:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:32:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:13.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:14.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:15.242 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:32:15.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:32:15.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:15.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:15.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:16.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:16.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:16.549 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:32:16.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:32:16.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:16.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:16.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:17.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:17.934 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:32:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:32:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:18.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:19.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:19.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:19.483 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:32:19.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:32:19.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:19.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:19.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:21.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:21.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:21.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:32:21.001 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:32:21.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:21.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:22.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:22.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:22.389 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:32:22.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:32:22.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:22.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:22.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:23.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:23.782 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:32:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:32:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:23.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:24.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:25.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:25.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:25.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:32:25.167 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:32:25.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:25.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:25.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:26.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:26.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:26.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:32:26.870 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:32:26.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:26.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:27.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:28.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:28.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:28.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:32:28.560 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:32:28.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:28.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:32:28.875 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2531277537,v1:127.0.0.1:6811/2531277537] [v2:127.0.0.1:6812/2531277537,v1:127.0.0.1:6813/2531277537] exists,up 2240e0a1-1fe3-451a-a329-2493ac8f29e7 2024-09-10T21:32:28.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:32:28.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:32:28.880 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:32:28.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:32:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:32:28.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:32:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=10c245ae-5276-4442-841c-4f67788e53ea 2024-09-10T21:32:28.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 10c245ae-5276-4442-841c-4f67788e53ea' 2024-09-10T21:32:28.882 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 10c245ae-5276-4442-841c-4f67788e53ea 2024-09-10T21:32:28.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:32:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsuuBmEeRtNRAAQe+9cCIJqIC5R0DnTUrQxA== 2024-09-10T21:32:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsuuBmEeRtNRAAQe+9cCIJqIC5R0DnTUrQxA=="}' 2024-09-10T21:32:28.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 10c245ae-5276-4442-841c-4f67788e53ea -i td/osd-scrub-repair/2/new.json 2024-09-10T21:32:29.223 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:32:29.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:32:29.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsuuBmEeRtNRAAQe+9cCIJqIC5R0DnTUrQxA== --osd-uuid 10c245ae-5276-4442-841c-4f67788e53ea 2024-09-10T21:32:29.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:29.262+0000 7f8022a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:29.299 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:29.296+0000 7f8022a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:29.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:29.298+0000 7f8022a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:29.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:29.298+0000 7f8022a1e640 -1 bdev(0x562242aa4800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:32:29.300 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:29.298+0000 7f8022a1e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:32:35.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:32:35.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:32:35.773 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:32:35.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:32:35.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:32:36.150 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:32:36.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:32:36.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:32:36.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:32:36.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:32:36.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:32:36.178 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:36.176+0000 7f13afa3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:36.180 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:36.178+0000 7f13afa3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:36.181 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:36.179+0000 7f13afa3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:32:36.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:32:36.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:32:36.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:32:36.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:32:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:32:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:32:36.426 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:32:36.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:36.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:36.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:37.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:37.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:37.703 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:32:37.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:32:37.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:37.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:37.793 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:37.792+0000 7f13afa3b640 -1 Falling back to public interface 2024-09-10T21:32:38.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:39.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:39.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:39.011 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:32:39.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:32:39.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:39.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:39.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:39.394 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:32:39.393+0000 7f13afa3b640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:32:40.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:40.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:40.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:32:40.299 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:32:40.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:40.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:40.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:41.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:41.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:41.630 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:32:41.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:32:41.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:41.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:41.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:42.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:42.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:42.961 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:32:42.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:32:42.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:42.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:43.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:44.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:44.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:44.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:32:44.301 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:32:44.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:44.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:44.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:45.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:45.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:45.591 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:32:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:32:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:45.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:46.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:46.880 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:32:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:32:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:47.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:48.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:48.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:48.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:32:48.266 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:32:48.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:48.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:48.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:49.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:49.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:49.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:32:49.671 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:32:49.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:49.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:50.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:51.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:51.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:51.002 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:32:51.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:32:51.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:51.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:51.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:52.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:52.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:52.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:32:52.325 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:32:52.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:52.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:52.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:53.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:32:53.886 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:32:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:53.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:54.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:55.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:55.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:55.738 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:32:55.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:32:55.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:55.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:56.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:57.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:57.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:57.148 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:32:57.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:32:57.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:57.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:57.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:58.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:58.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:58.595 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:32:58.595 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:32:58.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:58.596 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:32:58.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:32:59.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:32:59.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:32:59.969 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:32:59.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:32:59.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:32:59.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:00.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:01.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:01.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:01.688 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:33:01.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:33:01.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:01.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:02.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:03.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:03.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:03.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:33:03.310 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:33:03.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:03.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:03.597 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1033043685,v1:127.0.0.1:6819/1033043685] [v2:127.0.0.1:6820/1033043685,v1:127.0.0.1:6821/1033043685] exists,up 10c245ae-5276-4442-841c-4f67788e53ea 2024-09-10T21:33:03.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:33:03.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:33:03.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:33:03.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-10T21:33:03.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:33:03.865 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:33:03.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:33:03.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:33:04.252 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:33:04.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:33:05.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:33:07.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-10T21:33:07.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:33:07.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:33:07.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:33:07.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:33:07.108 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:33:07.108 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:33:07.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:33:07.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:33:07.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:33:07.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:33:07.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:33:07.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:33:07.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:33:07.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:33:07.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:33:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:33:07.490 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:07.491 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:33:07.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:33:07.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:07.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:33:07.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T21:33:07.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T21:33:07.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T21:33:07.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:07.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:33:07.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-10T21:33:07.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-10T21:33:07.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672971' 2024-09-10T21:33:07.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:07.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:33:07.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-10T21:33:07.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-10T21:33:07.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672971 2-64424509446' 2024-09-10T21:33:07.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:07.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T21:33:07.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:07.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:33:07.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T21:33:07.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:07.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T21:33:07.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T21:33:07.818 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836503 2024-09-10T21:33:07.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:33:08.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T21:33:08.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:08.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-10T21:33:08.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:08.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:33:08.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-10T21:33:08.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:08.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-10T21:33:08.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-10T21:33:08.095 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672971 2024-09-10T21:33:08.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:33:08.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672971 2024-09-10T21:33:08.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:33:09.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:33:09.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:33:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672971 2024-09-10T21:33:09.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:33:10.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:33:10.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:33:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672971 2024-09-10T21:33:10.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:10.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-10T21:33:10.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:10.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:33:10.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-10T21:33:10.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-10T21:33:10.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-10T21:33:10.952 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509446 2024-09-10T21:33:10.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:33:11.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-10T21:33:11.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:33:11.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:11.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:11.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:33:11.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:11.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:11.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:11.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:11.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:11.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:11.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:11.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:33:11.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:11.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:11.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:33:12.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T21:33:12.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:33:12.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T21:33:12.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:33:12.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:33:13.284 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:33:13.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:33:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T21:33:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:33:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:33:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:33:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:33:14.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:33:14.299 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:33:14.299 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:33:14.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:33:14.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:33:14.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:33:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:33:14.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:33:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:33:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:33:14.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:33:14.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:14.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:33:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T21:33:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T21:33:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T21:33:14.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:14.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:33:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-10T21:33:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-10T21:33:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672974' 2024-09-10T21:33:14.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:14.963 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:33:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-10T21:33:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-10T21:33:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672974 2-64424509449' 2024-09-10T21:33:15.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:15.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T21:33:15.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:15.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:33:15.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T21:33:15.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T21:33:15.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T21:33:15.070 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836506 2024-09-10T21:33:15.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:33:15.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-10T21:33:15.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:33:16.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:33:16.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:33:16.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T21:33:16.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-10T21:33:16.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:16.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-10T21:33:16.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-10T21:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-10T21:33:16.634 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672974 2024-09-10T21:33:16.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:33:16.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-10T21:33:16.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:16.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-10T21:33:16.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:16.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:33:16.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-10T21:33:16.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:16.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-10T21:33:16.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-10T21:33:16.912 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509449 2024-09-10T21:33:16.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:33:17.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-10T21:33:17.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:33:17.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:17.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:17.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:17.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:17.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:33:17.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:17.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:17.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:18.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:33:18.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T21:33:18.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:33:18.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T21:33:18.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:33:18.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:33:18.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:33:18.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:18.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:18.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:18.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:18.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:18.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:18.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:18.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:33:18.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:18.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:18.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:33:18.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:33:18.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:33:18.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:33:18.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:33:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:33:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:33:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:33:19.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:33:19.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:33:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:33:19.417 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:19.418 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:19.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:33:19.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:19.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:19.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:20.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:33:20.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:33:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:33:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:33:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T21:33:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:33:20.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T21:33:20.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:33:20.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:20.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:21.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:33:21.056 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:21.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:33:21.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:33:21.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-10T21:33:21.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:33:21.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:33:21.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:33:21.780 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:33:21.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:33:22.307 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:33:22.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:33:22.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:33:22.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:33:22.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-10T21:33:22.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:33:22.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-10T21:33:22.521 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-10T21:33:22.521 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:33:22.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:33:22.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:33:22.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:33:23.212 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:33:23.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:33:23.739 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:33:23.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:33:23.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:33:23.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:33:24.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-10T21:33:24.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:33:24.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-10T21:33:24.036 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-10T21:33:24.036 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-10T21:33:24.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-10T21:33:24.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-10T21:33:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:33:24.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:33:24.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:33:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:33:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:33:24.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:33:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:33:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:33:24.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:33:24.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:33:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:33:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:33:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:33:25.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:33:25.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:25.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:33:25.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:33:25.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:33:25.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:33:25.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:33:25.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:33:26.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:33:26.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-10T21:33:26.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-10T21:33:27.668 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-10T21:33:28.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-10T21:33:28.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-10T21:33:29.555 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-10T21:33:30.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-10T21:33:30.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-10T21:33:31.133 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-10T21:33:31.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-10T21:33:31.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-10T21:33:32.996 INFO:tasks.workunit.client.0.smithi181.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-10T21:33:33.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:33:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:33:33.519 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:33:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:33:33.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:33:33.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:33.545+0000 7f38f280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:33.549 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:33.548+0000 7f38f280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:33.550 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:33.549+0000 7f38f280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:33.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:33.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:33:34.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:34.902 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:34.900+0000 7f38f280e640 -1 Falling back to public interface 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:35.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:33:35.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:36.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:36.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:33:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:33:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:33:36.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:37.094 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:37.092+0000 7f38f280e640 -1 osd.0 33 log_to_monitors true 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:37.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:33:37.959 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/3083269767,v1:127.0.0.1:6803/3083269767] [v2:127.0.0.1:6804/3083269767,v1:127.0.0.1:6805/3083269767] exists,up fcebd4d9-d257-46df-b375-663e05d013df 2024-09-10T21:33:37.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:33:37.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:33:37.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:37.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:33:37.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:33:37.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:33:37.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:33:37.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:33:37.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:33:37.986 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:37.985+0000 7f71dca2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:37.988 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:37.987+0000 7f71dca2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:37.993 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:37.988+0000 7f71dca2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:33:38.284 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:33:38.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:38.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:33:38.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:39.337 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:39.336+0000 7f71dca2c640 -1 Falling back to public interface 2024-09-10T21:33:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:39.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:33:39.587 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:39.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:39.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:33:39.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:40.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:33:41.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:41.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:41.428+0000 7f71dca2c640 -1 osd.1 35 log_to_monitors true 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:42.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1598618542,v1:127.0.0.1:6811/1598618542] [v2:127.0.0.1:6812/1598618542,v1:127.0.0.1:6813/1598618542] exists,up 2240e0a1-1fe3-451a-a329-2493ac8f29e7 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-10T21:33:42.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:33:42.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:33:42.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:33:42.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:33:42.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:33:42.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:33:42.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:33:42.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:33:42.514 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:42.512+0000 7f1b747c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:42.591 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:42.589+0000 7f1b747c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:42.592 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:42.590+0000 7f1b747c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:42.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:43.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:43.942 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:43.941+0000 7f1b747c3640 -1 Falling back to public interface 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:44.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:44.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:45.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:45.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:33:45.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:33:45.933+0000 7f1b747c3640 -1 osd.2 37 log_to_monitors true 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:33:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/55051283,v1:127.0.0.1:6819/55051283] [v2:127.0.0.1:6820/55051283,v1:127.0.0.1:6821/55051283] exists,up 10c245ae-5276-4442-841c-4f67788e53ea 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-10T21:33:47.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:33:47.788 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' removed 2024-09-10T21:33:47.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:33:47.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:33:48.190 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:33:48.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:33:49.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:33:50.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-10T21:33:50.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:33:50.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:33:50.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:33:50.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:33:50.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:33:51.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:33:51.274 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:33:51.274 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:33:51.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:33:51.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:51.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:33:51.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691844 2024-09-10T21:33:51.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691844 2024-09-10T21:33:51.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691844' 2024-09-10T21:33:51.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:51.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:33:51.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593731 2024-09-10T21:33:51.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593731 2024-09-10T21:33:51.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-09-10T21:33:51.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:33:51.480 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:33:51.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462915 2024-09-10T21:33:51.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462915 2024-09-10T21:33:51.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-09-10T21:33:51.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:51.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691844 2024-09-10T21:33:51.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:51.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:33:51.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691844 2024-09-10T21:33:51.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:51.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691844 2024-09-10T21:33:51.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-09-10T21:33:51.585 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 171798691844 2024-09-10T21:33:51.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:33:51.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691844 2024-09-10T21:33:51.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:51.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593731 2024-09-10T21:33:51.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:51.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:33:51.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593731 2024-09-10T21:33:51.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:51.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593731 2024-09-10T21:33:51.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-09-10T21:33:51.915 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 184683593731 2024-09-10T21:33:51.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:33:52.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593732 -lt 184683593731 2024-09-10T21:33:52.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:33:52.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-201863462915 2024-09-10T21:33:52.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:33:52.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:33:52.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-201863462915 2024-09-10T21:33:52.233 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:33:52.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462915 2024-09-10T21:33:52.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-10T21:33:52.234 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 201863462915 2024-09-10T21:33:52.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:33:52.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-10T21:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:52.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:52.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:33:52.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:33:52.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:33:52.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:33:52.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:33:52.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:33:52.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:33:52.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:33:53.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:33:53.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:33:53.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:33:53.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:33:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:33:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:33:53.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:33:53.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-10T21:33:53.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:33:53.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-10T21:33:53.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-10T21:33:53.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:33:53.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-10T21:33:53.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-10T21:33:53.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:33:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:33:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:33:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:33:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:33:53.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:33:54.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:54.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:33:54.344 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:33:54.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:33:54.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:33:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:33:13.107192+0000 '>' 2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:54.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:33:55.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:33:55.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:33:55.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:33:55.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:33:55.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:33:55.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:33:55.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:33:55.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:33:13.107192+0000 '>' 2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:55.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:33:56.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:33:56.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:33:56.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:33:56.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:33:56.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:33:56.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:33:56.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:33:57.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:33:13.107192+0000 '>' 2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:57.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:33:58.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:33:58.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:33:58.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:33:58.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:33:58.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:33:58.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:33:58.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:33:58.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:33:54.807800+0000 '>' 2024-09-10T21:33:13.107192+0000 2024-09-10T21:33:58.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:33:58.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:34:00.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-10T21:34:00.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-10T21:34:00.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-10T21:34:00.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-10T21:34:00.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-10T21:34:00.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:34:00.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:34:00.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:34:00.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:34:00.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:34:00.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:34:00.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:34:00.773 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:34:00.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:34:00.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:34:00.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:34:00.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:34:00.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:34:00.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:34:00.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:34:00.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:34:00.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:34:00.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:34:00.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:34:00.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:00.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:00.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:34:00.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:34:00.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:34:00.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:34:00.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:34:00.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:34:00.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:34:00.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:34:00.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:34:00.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:34:00.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:34:00.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:34:00.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:34:00.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:34:00.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:34:00.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:34:00.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:34:00.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:34:00.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:34:00.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:00.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:34:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:34:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:34:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:34:00.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:34:00.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:00.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:00.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-09-10T21:34:00.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:34:00.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:34:01.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:34:01.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:34:01.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:34:01.252 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:34:01.252 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:01.253 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:01.253 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:01.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:34:01.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:34:01.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:34:01.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:34:01.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:34:01.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:34:01.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:34:01.438 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:34:01.438 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:34:01.438 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:34:01.438 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:34:01.438 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:01.439 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:01.439 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:34:01.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:34:01.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:34:01.518 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:34:01.519 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:34:01.519 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:01.519 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:01.519 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:34:01.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:34:01.520 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:34:01.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:34:02.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:02.405 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:34:02.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:34:02.434 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:34:02.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:34:02.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:34:02.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=564a5b31-e61a-4dda-a25e-8654be5aba6e 2024-09-10T21:34:02.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 564a5b31-e61a-4dda-a25e-8654be5aba6e' 2024-09-10T21:34:02.443 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 564a5b31-e61a-4dda-a25e-8654be5aba6e 2024-09-10T21:34:02.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:34:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKu+BmQFo7GxAA3z378h1c0ebCeIMyYkZZxQ== 2024-09-10T21:34:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKu+BmQFo7GxAA3z378h1c0ebCeIMyYkZZxQ=="}' 2024-09-10T21:34:02.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 564a5b31-e61a-4dda-a25e-8654be5aba6e -i td/osd-scrub-repair/0/new.json 2024-09-10T21:34:02.660 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:34:02.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:34:02.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKu+BmQFo7GxAA3z378h1c0ebCeIMyYkZZxQ== --osd-uuid 564a5b31-e61a-4dda-a25e-8654be5aba6e 2024-09-10T21:34:02.701 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:02.698+0000 7f85aefa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:02.712 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:02.709+0000 7f85aefa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:02.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:02.711+0000 7f85aefa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:02.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:02.711+0000 7f85aefa2640 -1 bdev(0x5644c5d1dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:34:02.713 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:02.711+0000 7f85aefa2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:34:08.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:34:08.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:34:08.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:34:08.436 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:34:08.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:34:08.792 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:34:08.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:34:08.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:34:08.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:34:08.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:34:08.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:34:08.820 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:08.817+0000 7f0694c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:08.822 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:08.819+0000 7f0694c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:08.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:08.820+0000 7f0694c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:34:09.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:09.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:34:09.069 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:34:09.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:09.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:09.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:10.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:10.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:10.346 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:34:10.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:34:10.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:10.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:10.441 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:10.439+0000 7f0694c1c640 -1 Falling back to public interface 2024-09-10T21:34:10.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:11.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:11.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:11.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:34:11.637 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:34:11.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:11.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:11.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:12.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:12.390+0000 7f0694c1c640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:34:12.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:12.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:12.925 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:34:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:34:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:12.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:13.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:14.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:34:14.298 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:34:14.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:14.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:14.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:15.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:15.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:15.589 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:34:15.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:34:15.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:15.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:15.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:16.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:16.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:16.912 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:34:16.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:34:16.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:16.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:17.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:18.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:18.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:18.203 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:34:18.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:34:18.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:18.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:18.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:19.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:19.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:19.490 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:19.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:19.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:20.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:20.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:20.778 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:34:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:34:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:20.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:21.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:22.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:22.152 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:34:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:34:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:22.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:22.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:23.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:23.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:23.443 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:34:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:34:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:23.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:24.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:24.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:24.734 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:34:24.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:34:24.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:24.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:25.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:26.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:26.305 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:34:26.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:34:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:26.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:27.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:28.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:28.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:28.087 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:34:28.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:34:28.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:28.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:28.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:29.726 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:34:29.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:34:29.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:29.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:30.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:31.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:31.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:31.215 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:34:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:34:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:31.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:31.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:32.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:32.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:32.504 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:34:32.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:34:32.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:32.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:33.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:34.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:34.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:34.251 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:34:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:34:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:34.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:34:34.596 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2329525906,v1:127.0.0.1:6803/2329525906] [v2:127.0.0.1:6804/2329525906,v1:127.0.0.1:6805/2329525906] exists,up 564a5b31-e61a-4dda-a25e-8654be5aba6e 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:34:34.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:34:34.600 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:34:34.601 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:34:34.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:34:34.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:34:34.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:34:34.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:34:34.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=35cb387e-8f06-42de-baee-1ebda610a636 2024-09-10T21:34:34.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 35cb387e-8f06-42de-baee-1ebda610a636' 2024-09-10T21:34:34.613 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 35cb387e-8f06-42de-baee-1ebda610a636 2024-09-10T21:34:34.613 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:34:34.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqu+BmGs2CJRAA2u8F/Aih9eEuhVWZYKgwrA== 2024-09-10T21:34:34.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqu+BmGs2CJRAA2u8F/Aih9eEuhVWZYKgwrA=="}' 2024-09-10T21:34:34.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 35cb387e-8f06-42de-baee-1ebda610a636 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:34:34.953 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:34:34.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:34:34.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqu+BmGs2CJRAA2u8F/Aih9eEuhVWZYKgwrA== --osd-uuid 35cb387e-8f06-42de-baee-1ebda610a636 2024-09-10T21:34:34.996 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:34.994+0000 7f4511c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:35.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:35.036+0000 7f4511c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:35.038 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:35.037+0000 7f4511c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:35.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:35.038+0000 7f4511c20640 -1 bdev(0x55d7bae6dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:34:35.039 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:35.038+0000 7f4511c20640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:34:41.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:34:41.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:34:41.402 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:34:41.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:34:41.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:34:41.756 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:34:41.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:34:41.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:34:41.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:34:41.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:34:41.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:34:41.784 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:41.782+0000 7ff02214d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:41.786 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:41.784+0000 7ff02214d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:41.787 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:41.785+0000 7ff02214d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:34:42.034 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:34:42.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:42.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:42.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:43.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:43.148+0000 7ff02214d640 -1 Falling back to public interface 2024-09-10T21:34:43.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:43.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:43.313 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:34:43.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:34:43.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:43.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:43.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:44.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:44.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:44.594 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:34:44.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:34:44.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:44.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:44.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:45.127 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:34:45.126+0000 7ff02214d640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:34:45.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:45.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:45.882 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:34:45.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:34:45.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:45.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:46.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:47.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:47.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:47.160 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:34:47.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:34:47.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:47.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:47.444 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:48.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:48.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:48.446 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:34:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:34:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:49.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:49.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:49.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:34:49.812 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:34:49.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:49.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:50.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:51.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:51.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:51.142 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:34:51.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:34:51.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:51.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:51.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:52.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:52.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:52.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:34:52.430 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:34:52.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:52.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:52.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:53.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:53.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:53.720 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:34:53.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:34:53.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:53.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:54.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:55.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:55.006 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:34:55.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:34:55.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:55.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:56.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:56.359 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:34:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:34:56.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:56.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:56.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:57.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:57.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:34:57.651 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:34:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:57.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:58.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:34:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:34:59.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:34:59.373 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:34:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:34:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:34:59.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:34:59.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:00.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:00.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:00.886 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:35:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:35:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:00.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:01.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:02.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:02.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:02.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:35:02.552 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:35:02.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:03.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:04.221 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:35:04.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:35:04.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:04.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:04.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:05.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:05.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:05.940 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:35:05.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:35:05.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:05.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:06.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:07.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:07.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:07.546 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:35:07.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:35:07.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:07.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:07.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:08.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:08.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:08.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:35:08.980 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:35:08.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:08.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:35:09.269 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1592259802,v1:127.0.0.1:6811/1592259802] [v2:127.0.0.1:6812/1592259802,v1:127.0.0.1:6813/1592259802] exists,up 35cb387e-8f06-42de-baee-1ebda610a636 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:35:09.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:35:09.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:35:09.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=44e1d7eb-0774-499b-bdb6-a9449ca96ea3 2024-09-10T21:35:09.276 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 44e1d7eb-0774-499b-bdb6-a9449ca96ea3 2024-09-10T21:35:09.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 44e1d7eb-0774-499b-bdb6-a9449ca96ea3' 2024-09-10T21:35:09.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:35:09.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCNu+BmxzlLERAAqi2BcJdE3aoqDk80bCAkPA== 2024-09-10T21:35:09.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCNu+BmxzlLERAAqi2BcJdE3aoqDk80bCAkPA=="}' 2024-09-10T21:35:09.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 44e1d7eb-0774-499b-bdb6-a9449ca96ea3 -i td/osd-scrub-repair/2/new.json 2024-09-10T21:35:09.653 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:35:09.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:35:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNu+BmxzlLERAAqi2BcJdE3aoqDk80bCAkPA== --osd-uuid 44e1d7eb-0774-499b-bdb6-a9449ca96ea3 2024-09-10T21:35:09.694 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:09.693+0000 7fba23efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:09.737 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:09.736+0000 7fba23efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:09.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:09.737+0000 7fba23efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:09.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:09.738+0000 7fba23efe640 -1 bdev(0x557d3ed91c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:35:09.739 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:09.738+0000 7fba23efe640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:35:15.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:35:15.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:35:15.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:35:15.747 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:35:15.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:35:16.104 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:35:16.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:35:16.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:35:16.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:35:16.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:35:16.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:35:16.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:16.130+0000 7fee56211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:16.133 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:16.132+0000 7fee56211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:16.134 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:16.133+0000 7fee56211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:16.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:35:16.384 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:35:16.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:16.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:16.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:17.272 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:17.271+0000 7fee56211640 -1 Falling back to public interface 2024-09-10T21:35:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:17.710 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:35:17.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:35:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:17.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:17.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:18.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:18.999 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:35:19.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:35:19.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:19.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:19.103 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:35:19.102+0000 7fee56211640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:35:19.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:20.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:20.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:20.330 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:35:20.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:35:20.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:20.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:20.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:21.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:35:21.669 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:35:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:21.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:22.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:22.958 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:22.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:23.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:24.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:24.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:24.345 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:35:24.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:35:24.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:24.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:24.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:25.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:25.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:25.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:35:25.677 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:35:25.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:25.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:25.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:26.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:26.969 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:35:26.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:35:26.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:26.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:27.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:28.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:28.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:28.364 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:35:28.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:35:28.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:28.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:28.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:29.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:29.702 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:35:29.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:29.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:35:29.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:29.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:30.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:31.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:31.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:31.101 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:35:31.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:35:31.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:31.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:31.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:32.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:32.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:32.544 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:35:32.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:35:32.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:32.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:33.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:34.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:34.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:34.324 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:35:34.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:35:34.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:34.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:34.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:35.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:35.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:35.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:35:35.774 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:35:35.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:35.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:36.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:35:37.435 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:35:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:37.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:37.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:38.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:38.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:38.972 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:35:38.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:35:38.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:38.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:39.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:40.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:40.725 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:35:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:35:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:40.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:41.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:35:42.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:35:42.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:35:42.134 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:35:42.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:35:42.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:35:42.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:35:42.421 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3502262901,v1:127.0.0.1:6819/3502262901] [v2:127.0.0.1:6820/3502262901,v1:127.0.0.1:6821/3502262901] exists,up 44e1d7eb-0774-499b-bdb6-a9449ca96ea3 2024-09-10T21:35:42.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:35:42.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:35:42.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:35:42.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-10T21:35:42.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:35:42.745 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:35:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:35:42.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:35:43.136 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:35:43.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:35:44.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:35:46.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-10T21:35:46.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:35:46.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:35:46.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:35:46.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:35:46.863 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:35:46.863 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:35:46.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:35:46.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:35:46.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:35:46.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:35:46.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:35:46.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:35:46.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:35:46.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:35:46.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:47.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:35:47.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-10T21:35:47.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-10T21:35:47.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-10T21:35:47.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:47.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:35:47.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-10T21:35:47.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-10T21:35:47.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672972' 2024-09-10T21:35:47.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:47.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:35:47.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-10T21:35:47.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-10T21:35:47.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672972 2-64424509446' 2024-09-10T21:35:47.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:47.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-10T21:35:47.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:47.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:35:47.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-10T21:35:47.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:47.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-10T21:35:47.559 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-10T21:35:47.560 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836499 2024-09-10T21:35:47.560 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:35:47.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836499 2024-09-10T21:35:47.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:35:48.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:35:48.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:35:49.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-10T21:35:49.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:49.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-10T21:35:49.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:49.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:35:49.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-10T21:35:49.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:49.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-10T21:35:49.122 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672972 2024-09-10T21:35:49.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-10T21:35:49.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:35:49.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-10T21:35:49.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:49.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-10T21:35:49.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:49.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:35:49.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-10T21:35:49.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:49.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-10T21:35:49.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-10T21:35:49.401 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509446 2024-09-10T21:35:49.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:35:49.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-10T21:35:49.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:35:49.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:35:49.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:35:50.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:35:50.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:35:50.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:35:50.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:35:50.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:35:50.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:35:50.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:35:50.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:35:50.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:35:50.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:35:50.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:35:50.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:35:50.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T21:35:51.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:35:51.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:35:51.877 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:35:51.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:35:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T21:35:52.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T21:35:53.476 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 2 allow_ec_overwrites to true 2024-09-10T21:35:53.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:35:53.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:35:53.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:35:53.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:35:53.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:35:53.489 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:35:53.489 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:35:53.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:35:53.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:35:53.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:35:53.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:53.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:35:54.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T21:35:54.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T21:35:54.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T21:35:54.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:54.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:35:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-10T21:35:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-10T21:35:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-09-10T21:35:54.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:35:54.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:35:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-10T21:35:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-10T21:35:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509449' 2024-09-10T21:35:54.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:54.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T21:35:54.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:54.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:35:54.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T21:35:54.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T21:35:54.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T21:35:54.356 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T21:35:54.357 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:35:54.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-10T21:35:54.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:54.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-10T21:35:54.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:54.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:35:54.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-10T21:35:54.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:54.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-10T21:35:54.657 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672975 2024-09-10T21:35:54.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-10T21:35:54.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:35:54.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-10T21:35:54.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:35:54.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-10T21:35:54.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:35:54.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:35:54.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-10T21:35:54.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:35:54.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-10T21:35:54.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-10T21:35:54.937 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509449 2024-09-10T21:35:54.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:35:55.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-10T21:35:55.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:35:55.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:35:55.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:35:55.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:35:55.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:35:55.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:35:55.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:35:55.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:35:55.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:35:55.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:35:55.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:35:55.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:35:55.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:35:55.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:35:55.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:35:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:35:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:35:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:35:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:35:56.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:35:56.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:35:56.576 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:35:56.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:35:56.994 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:35:57.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:35:57.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:35:57.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:35:57.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-10T21:35:57.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:35:57.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-10T21:35:57.284 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-10T21:35:57.284 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:35:57.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T21:35:57.564 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:35:57.564 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:35:57.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:35:57.932 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:35:57.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:35:58.476 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:35:58.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:35:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:35:58.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:35:58.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-10T21:35:58.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:35:58.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-10T21:35:58.781 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-10T21:35:58.782 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:0' 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-10T21:35:59.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-10T21:35:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:35:59.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:35:59.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:35:59.084 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:35:59.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:35:59.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:35:59.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:35:59.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:35:59.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:35:59.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:35:59.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:35:59.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:35:59.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:35:59.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:35:59.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:35:59.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:35:59.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:35:59.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:35:59.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:35:59.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:35:59.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:35:59.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:35:59.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:36:01.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:36:01.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-10T21:36:01.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-10T21:36:02.375 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-10T21:36:02.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-10T21:36:02.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-10T21:36:04.246 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-10T21:36:04.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-10T21:36:04.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-10T21:36:05.800 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-10T21:36:06.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-10T21:36:06.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-10T21:36:07.653 INFO:tasks.workunit.client.0.smithi181.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-10T21:36:08.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-10T21:36:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:36:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-10T21:36:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:36:08.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:36:08.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:36:08.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:36:08.175 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:36:08.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:36:08.204 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:08.202+0000 7ff61fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:08.206 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:08.205+0000 7ff61fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:08.207 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:08.206+0000 7ff61fc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:36:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:36:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:36:08.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:08.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:09.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:09.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:09.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:36:09.736 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:36:09.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:09.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:09.808 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:09.807+0000 7ff61fc2c640 -1 Falling back to public interface 2024-09-10T21:36:10.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:11.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:11.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:11.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:11.453+0000 7ff61fc2c640 -1 osd.0 36 log_to_monitors true 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:12.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/905002686,v1:127.0.0.1:6803/905002686] [v2:127.0.0.1:6804/905002686,v1:127.0.0.1:6805/905002686] exists,up 564a5b31-e61a-4dda-a25e-8654be5aba6e 2024-09-10T21:36:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:36:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:36:12.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:36:12.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:36:12.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.1 2024-09-10T21:36:12.719 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:36:12.720 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:36:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:36:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:36:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:36:12.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:36:12.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:12.742+0000 7f2861fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:12.746 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:12.745+0000 7f2861fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:12.747 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:12.746+0000 7f2861fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:36:13.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:13.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:36:13.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:14.093 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:14.092+0000 7f2861fb2640 -1 Falling back to public interface 2024-09-10T21:36:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:36:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:36:14.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:14.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:36:14.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:15.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:36:15.627 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:15.626+0000 7f2861fb2640 -1 osd.1 37 log_to_monitors true 2024-09-10T21:36:15.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:16.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/3115432454,v1:127.0.0.1:6811/3115432454] [v2:127.0.0.1:6812/3115432454,v1:127.0.0.1:6813/3115432454] exists,up 35cb387e-8f06-42de-baee-1ebda610a636 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:36:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:36:17.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:36:17.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.2 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:36:17.207 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:36:17.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:36:17.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:36:17.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:36:17.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:36:17.231 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:17.229+0000 7f6e458ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:17.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:17.231+0000 7f6e458ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:17.234 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:17.232+0000 7f6e458ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:36:17.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:36:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:36:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:17.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:36:17.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:18.353 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:18.352+0000 7f6e458ef640 -1 Falling back to public interface 2024-09-10T21:36:18.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:18.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:18.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:36:18.759 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:36:18.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:18.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:36:19.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:20.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:20.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:20.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:36:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:36:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:20.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:36:20.196 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:20.195+0000 7f6e458ef640 -1 osd.2 38 log_to_monitors true 2024-09-10T21:36:20.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:36:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:36:21.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:21.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:36:21.713 INFO:tasks.workunit.client.0.smithi181.stderr:osd.2 up in weight 1 up_from 48 up_thru 48 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/1983214338,v1:127.0.0.1:6819/1983214338] [v2:127.0.0.1:6820/1983214338,v1:127.0.0.1:6821/1983214338] exists,up 44e1d7eb-0774-499b-bdb6-a9449ca96ea3 2024-09-10T21:36:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:36:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:36:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:36:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-10T21:36:21.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:36:22.200 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' removed 2024-09-10T21:36:22.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:36:22.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:36:22.702 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:36:22.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:36:23.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:36:25.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-10T21:36:25.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:36:25.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:36:25.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:36:25.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:36:25.039 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:36:25.039 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:36:25.039 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:36:25.040 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:36:25.040 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:36:25.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:36:25.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:36:25.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:36:25.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:36:25.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:36:25.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:36:25.422 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:36:25.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659140 2024-09-10T21:36:25.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659140 2024-09-10T21:36:25.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659140' 2024-09-10T21:36:25.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:36:25.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:36:25.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561027 2024-09-10T21:36:25.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561027 2024-09-10T21:36:25.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-09-10T21:36:25.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:36:25.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:36:25.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-10T21:36:25.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-10T21:36:25.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-206158430211' 2024-09-10T21:36:25.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:36:25.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-176093659140 2024-09-10T21:36:25.734 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:36:25.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:36:25.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-176093659140 2024-09-10T21:36:25.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:36:25.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659140 2024-09-10T21:36:25.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-09-10T21:36:25.736 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 176093659140 2024-09-10T21:36:25.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:36:26.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659139 -lt 176093659140 2024-09-10T21:36:26.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:36:27.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:36:27.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:36:27.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659140 -lt 176093659140 2024-09-10T21:36:27.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:36:27.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561027 2024-09-10T21:36:27.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:36:27.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:36:27.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561027 2024-09-10T21:36:27.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:36:27.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561027 2024-09-10T21:36:27.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-09-10T21:36:27.295 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 188978561027 2024-09-10T21:36:27.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:36:27.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561028 -lt 188978561027 2024-09-10T21:36:27.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:36:27.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-206158430211 2024-09-10T21:36:27.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:36:27.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:36:27.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-206158430211 2024-09-10T21:36:27.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:36:27.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-10T21:36:27.574 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 206158430211 2024-09-10T21:36:27.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-10T21:36:27.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:36:27.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-10T21:36:27.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:36:27.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:36:27.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:36:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:36:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:36:28.218 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:36:28.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:36:28.493 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:36:28.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:36:28.494 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:36:28.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-10T21:36:28.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:36:29.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:36:29.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:36:29.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:29.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-10T21:36:29.713 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-10T21:36:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-10T21:36:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:36:29.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:36:29.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:36:29.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:35:51.675143+0000 '>' 2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:29.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:36:30.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:36:30.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:36:30.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:36:30.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:36:30.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:36:30.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:36:30.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:36:31.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:35:51.675143+0000 '>' 2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:31.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:36:32.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:36:32.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:36:32.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-10T21:36:32.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-10T21:36:32.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:36:32.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:36:32.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-10T21:36:32.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:36:30.083791+0000 '>' 2024-09-10T21:35:51.675143+0000 2024-09-10T21:36:32.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:36:32.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:36:34.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-10T21:36:34.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-10T21:36:34.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-10T21:36:34.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-10T21:36:34.570 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-10T21:36:34.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-10T21:36:34.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-10T21:36:34.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-10T21:36:34.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-10T21:36:34.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:36:34.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:36:34.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:36:34.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:36:34.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:36:34.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:36:34.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:36:34.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:36:34.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:36:34.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:36:34.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:36:34.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:36:34.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:36:34.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:36:34.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:36:34.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:36:34.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:36:34.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.820 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:36:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:36:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:36:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:36:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:36:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:36:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:36:34.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:36:34.824 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:36:34.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:36:34.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:36:34.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:36:34.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:36:34.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:36:34.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:36:34.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:36:34.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:36:34.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:36:34.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:36:34.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:36:34.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:36:34.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:36:34.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:36:34.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:36:34.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:36:34.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:36:34.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5762: TEST_periodic_scrub_replicated: local objname=POBJ 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: return 0 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:36:34.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:36:34.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:36:34.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:36:34.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:36:34.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:36:34.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:36:34.842 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:36:34.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:36:34.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:36:34.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:36:34.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:36:34.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:36:34.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:36:34.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:36:34.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:36:34.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:36:34.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:36:34.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:36:34.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:36:34.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:36:34.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:36:34.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:36:34.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:36:34.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:36:34.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:36:34.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:36:34.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:36:34.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:36:34.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:36:34.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:36:34.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:36:34.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:36:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:36:34.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:36:34.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:36:34.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:34.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:34.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-09-10T21:36:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:36:34.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:35.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:35.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:36:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:36:35.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:36:35.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:36:35.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:36:35.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:36:35.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:36:35.533 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:36:35.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:36:35.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:35.619 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:36:35.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:36:35.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:36:35.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:36.500 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:36.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:36:36.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:36:36.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:36:36.531 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-09-10T21:36:36.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:36:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:36:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:36:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:36:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:36:36.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:36:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2bdd5acf-4c77-49d8-9393-5460f86a05e4 2024-09-10T21:36:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2bdd5acf-4c77-49d8-9393-5460f86a05e4' 2024-09-10T21:36:36.540 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 2bdd5acf-4c77-49d8-9393-5460f86a05e4 2024-09-10T21:36:36.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:36:36.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDku+Bmgr4OIRAAfeg0XRht0scDTrGfl4F0tA== 2024-09-10T21:36:36.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDku+Bmgr4OIRAAfeg0XRht0scDTrGfl4F0tA=="}' 2024-09-10T21:36:36.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2bdd5acf-4c77-49d8-9393-5460f86a05e4 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:36:36.774 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:36:36.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:36:36.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDku+Bmgr4OIRAAfeg0XRht0scDTrGfl4F0tA== --osd-uuid 2bdd5acf-4c77-49d8-9393-5460f86a05e4 2024-09-10T21:36:36.814 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:36.812+0000 7f15d55c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:36.823 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:36.821+0000 7f15d55c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:36.824 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:36.822+0000 7f15d55c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:36.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:36.822+0000 7f15d55c3640 -1 bdev(0x5556461e3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:36:36.825 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:36.822+0000 7f15d55c3640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:36:42.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:36:42.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:36:42.297 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:36:42.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:36:42.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:36:42.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:36:42.663 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:36:42.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:36:42.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:36:42.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:36:42.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:36:42.692 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:42.690+0000 7fe9ece0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:42.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:42.718+0000 7fe9ece0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:42.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:42.720+0000 7fe9ece0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:36:42.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:36:42.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:36:42.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:42.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:43.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:44.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:44.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:44.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:36:44.223 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:36:44.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:44.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:44.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:44.331+0000 7fe9ece0d640 -1 Falling back to public interface 2024-09-10T21:36:44.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:45.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:45.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:45.504 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:36:45.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:36:45.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:45.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:45.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:46.237 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:36:46.235+0000 7fe9ece0d640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:36:46.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:46.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:46.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:36:46.789 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:36:46.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:46.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:47.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:48.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:48.180 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:36:48.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:48.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:36:48.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:48.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:48.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:49.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:49.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:49.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:36:49.469 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:36:49.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:49.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:49.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:50.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:50.776 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:36:50.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:50.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:36:50.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:50.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:51.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:52.065 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:36:52.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:36:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:52.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:52.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:53.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:53.352 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:36:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:36:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:53.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:53.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:54.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:54.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:54.640 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:36:54.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:36:54.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:54.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:55.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:56.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:56.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:56.034 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:36:56.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:36:56.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:56.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:56.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:57.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:57.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:57.322 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:36:57.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:36:57.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:57.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:57.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:36:58.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:36:58.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:36:58.610 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:36:58.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:36:58.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:36:58.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:36:59.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:00.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:00.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:00.186 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:37:00.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:37:00.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:00.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:01.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:02.120 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:37:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:37:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:02.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:02.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:03.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:03.555 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:37:03.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:03.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:37:03.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:03.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:04.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:05.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:37:05.317 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:37:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:05.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:06.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:07.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:07.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:07.102 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:37:07.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:37:07.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:07.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:07.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:08.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:08.805 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:37:08.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:08.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:37:08.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:08.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:37:09.090 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3266591299,v1:127.0.0.1:6803/3266591299] [v2:127.0.0.1:6804/3266591299,v1:127.0.0.1:6805/3266591299] exists,up 2bdd5acf-4c77-49d8-9393-5460f86a05e4 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:37:09.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:37:09.096 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:37:09.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5540da7c-00c5-48ce-8cc8-647f06640ef2 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5540da7c-00c5-48ce-8cc8-647f06640ef2' 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 5540da7c-00c5-48ce-8cc8-647f06640ef2 2024-09-10T21:37:09.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:37:09.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFvOBmzjKyBhAAkC5IZcihtgHRegeovCKIPg== 2024-09-10T21:37:09.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFvOBmzjKyBhAAkC5IZcihtgHRegeovCKIPg=="}' 2024-09-10T21:37:09.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5540da7c-00c5-48ce-8cc8-647f06640ef2 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:37:09.539 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:37:09.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:37:09.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAFvOBmzjKyBhAAkC5IZcihtgHRegeovCKIPg== --osd-uuid 5540da7c-00c5-48ce-8cc8-647f06640ef2 2024-09-10T21:37:09.583 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:09.581+0000 7f330760b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:09.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:09.623+0000 7f330760b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:09.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:09.624+0000 7f330760b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:09.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:09.624+0000 7f330760b640 -1 bdev(0x55dd6aa1bc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:37:09.625 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:09.624+0000 7f330760b640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:37:15.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:37:15.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:37:15.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:37:15.864 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:37:15.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:37:16.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:37:16.221 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:37:16.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:37:16.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:37:16.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:37:16.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:37:16.249 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:16.248+0000 7f66a640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:16.251 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:16.250+0000 7f66a640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:16.252 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:16.251+0000 7f66a640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:16.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:37:16.499 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:37:16.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:16.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:16.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:17.618 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:17.617+0000 7f66a640e640 -1 Falling back to public interface 2024-09-10T21:37:17.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:17.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:17.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:37:17.801 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:37:17.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:17.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:18.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:19.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:19.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:19.083 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:19.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:19.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:37:19.235+0000 7f66a640e640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:37:19.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:20.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:20.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:20.447 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:37:20.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:37:20.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:20.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:20.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:21.801 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:37:21.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:37:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:21.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:22.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:23.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:23.209 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:37:23.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:37:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:23.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:23.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:24.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:24.537 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:37:24.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:24.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:37:24.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:24.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:24.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:25.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:25.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:25.827 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:37:25.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:37:25.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:25.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:26.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:27.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:27.238 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:37:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:37:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:27.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:27.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:28.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:28.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:28.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:37:28.636 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:37:28.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:28.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:28.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:29.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:29.945 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:37:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:37:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:29.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:30.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:31.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:31.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:31.233 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:37:31.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:37:31.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:31.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:31.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:32.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:32.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:32.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:37:32.524 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:37:32.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:32.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:33.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:34.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:34.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:34.344 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:37:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:37:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:34.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:35.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:36.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:36.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:36.083 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:37:36.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:37:36.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:36.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:36.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:37.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:37.642 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:37:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:37:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:37.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:37.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:38.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:38.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:38.936 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:37:38.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:37:38.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:38.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:39.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:40.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:40.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:40.321 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:37:40.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:37:40.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:40.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:40.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:41.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:37:41.742 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:37:41.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:41.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:42.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:37:43.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:37:43.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:37:43.102 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:37:43.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:37:43.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:37:43.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:37:43.376 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/123380378,v1:127.0.0.1:6811/123380378] [v2:127.0.0.1:6812/123380378,v1:127.0.0.1:6813/123380378] exists,up 5540da7c-00c5-48ce-8cc8-647f06640ef2 2024-09-10T21:37:43.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:37:43.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:37:43.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:37:43.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-09-10T21:37:43.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-10T21:37:43.743 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T21:37:43.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:37:44.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-09-10T21:37:45.290 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 size to 2 2024-09-10T21:37:45.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-09-10T21:37:45.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:37:45.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:37:45.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:37:45.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:37:45.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:37:45.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:37:45.709 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:37:45.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:37:45.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:37:45.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:37:45.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-10T21:37:45.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-10T21:37:45.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-10T21:37:45.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:37:45.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:37:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-10T21:37:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-10T21:37:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-09-10T21:37:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:37:45.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-10T21:37:45.926 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:37:45.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:37:45.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-10T21:37:45.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:37:45.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-10T21:37:45.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-10T21:37:45.929 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836494 2024-09-10T21:37:45.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:37:46.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836494 2024-09-10T21:37:46.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:37:47.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:37:47.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:37:47.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-10T21:37:47.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:37:47.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-10T21:37:47.484 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:37:47.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:37:47.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-10T21:37:47.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:37:47.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-10T21:37:47.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-10T21:37:47.487 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672966 2024-09-10T21:37:47.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:37:47.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-10T21:37:47.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:37:47.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:37:47.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:37:48.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:37:48.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:37:48.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:37:48.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:37:48.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:37:48.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:37:48.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-09-10T21:37:48.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-09-10T21:37:48.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:48.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:48.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:48.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:49.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:49.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:50.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:50.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:51.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:51.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-10T21:37:52.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T21:37:52.954 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:37:53.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:37:53.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-10T21:37:53.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:37:53.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-09-10T21:37:53.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-09-10T21:37:53.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T21:37:53.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-10T21:37:53.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-10T21:37:53.512 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:37:53.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-09-10T21:37:53.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-09-10T21:37:53.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T21:37:53.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:37:53.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T21:37:53.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:37:54.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-09-10T21:37:54.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:37:54.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:37:54.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:37:54.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:37:54.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:37:54.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:37:54.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:37:54.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:37:54.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:37:54.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:37:54.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:37:54.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:37:54.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:37:54.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:37:54.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-09-10T21:37:54.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:37:54.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-09-10T21:37:54.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:37:54.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-10T21:37:56.046 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T21:37:56.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:37:56.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-09-10T21:37:56.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:37:56.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-10T21:37:57.918 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:104778fc:::obj2:head# 2024-09-10T21:37:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:37:58.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-09-10T21:37:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:37:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-10T21:37:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:37:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:37:58.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:37:58.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:37:58.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:37:58.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-10T21:37:59.470 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:8dd16f86:::obj3:head# 2024-09-10T21:37:59.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:37:59.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-09-10T21:37:59.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:37:59.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:37:59.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-10T21:38:01.147 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:0ee9ae15:::obj4:head# 2024-09-10T21:38:01.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:01.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:01.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-10T21:38:02.657 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:head# 2024-09-10T21:38:03.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:03.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-09-10T21:38:03.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:03.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-10T21:38:03.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:03.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:03.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:03.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:03.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:03.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-10T21:38:04.128 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:233a42c1:::obj6:head# 2024-09-10T21:38:04.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:04.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-09-10T21:38:04.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:04.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-10T21:38:04.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:04.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:04.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:04.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:04.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:04.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-10T21:38:05.588 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:9a09113a:::obj7:head# 2024-09-10T21:38:06.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:06.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:06.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-10T21:38:07.048 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:a557efb1:::obj8:head# 2024-09-10T21:38:07.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:07.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-09-10T21:38:07.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:07.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-10T21:38:08.533 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:fe351e27:::obj9:head# 2024-09-10T21:38:09.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:09.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:09.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-10T21:38:09.961 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:d1337354:::obj10:head# 2024-09-10T21:38:10.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:10.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-09-10T21:38:10.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-10T21:38:11.663 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:eeae2a94:::obj11:head# 2024-09-10T21:38:12.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:12.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:12.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-10T21:38:13.367 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:af5f95cb:::obj12:head# 2024-09-10T21:38:14.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:14.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-10T21:38:15.295 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:a61ad63e:::obj13:head# 2024-09-10T21:38:15.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:15.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-09-10T21:38:15.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:15.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:15.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-10T21:38:16.813 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:e0a44829:::obj14:head# 2024-09-10T21:38:17.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:17.328 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:17.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-10T21:38:18.232 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ab946124:::obj15:head# 2024-09-10T21:38:18.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:18.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-09-10T21:38:18.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:18.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-10T21:38:19.650 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:461f8b5e:::obj16:head# 2024-09-10T21:38:20.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:20.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-09-10T21:38:20.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:20.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-10T21:38:20.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:20.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:20.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:20.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:20.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:20.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-10T21:38:21.178 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:24fc6e92:::obj17:head# 2024-09-10T21:38:21.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:21.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-09-10T21:38:21.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:21.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-10T21:38:21.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:21.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:21.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:21.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:21.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:21.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-10T21:38:22.712 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:b9836a99:::obj18:head# 2024-09-10T21:38:23.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:23.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-09-10T21:38:23.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-10T21:38:23.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-10T21:38:23.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:23.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:23.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:38:23.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:23.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:23.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-10T21:38:24.406 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:00b31b6c:::obj19:head# 2024-09-10T21:38:24.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-10T21:38:24.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:24.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-10T21:38:25.859 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:24b5611b:::obj20:head# 2024-09-10T21:38:26.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:38:26.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:38:26.378 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:38:26.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:38:26.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:38:26.380 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:38:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:38:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:38:26.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:38:26.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:38:26.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:38:26.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:38:26.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:38:26.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:26.407+0000 7f23d85dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:26.411 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:26.409+0000 7f23d85dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:26.413 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:26.411+0000 7f23d85dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:38:26.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:26.667 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:38:26.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:38:26.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:26.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:38:26.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:38:27.537 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:27.535+0000 7f23d85dd640 -1 Falling back to public interface 2024-09-10T21:38:27.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:38:27.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:27.948 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:38:27.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:38:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:27.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:38:28.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:38:29.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:38:29.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:29.235 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:38:29.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:38:29.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:29.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:38:29.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:38:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:29.522+0000 7f23d85dd640 -1 osd.1 16 log_to_monitors true 2024-09-10T21:38:30.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:38:30.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:30.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:38:30.523 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:38:30.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:30.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:38:30.828 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 19 up_thru 19 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6802/3311644073,v1:127.0.0.1:6803/3311644073] [v2:127.0.0.1:6804/3311644073,v1:127.0.0.1:6805/3311644073] exists,up 5540da7c-00c5-48ce-8cc8-647f06640ef2 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:38:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:38:30.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:38:30.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:38:30.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:38:30.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:38:30.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:38:30.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:38:30.835 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:38:30.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:38:30.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:38:30.860 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:30.859+0000 7fb9d4815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:30.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:30.862+0000 7fb9d4815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:30.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:30.863+0000 7fb9d4815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:31.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:38:31.153 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:38:31.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:31.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:38:31.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:38:31.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:31.747+0000 7fb9d4815640 -1 Falling back to public interface 2024-09-10T21:38:32.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:38:32.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:32.430 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:38:32.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:38:32.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:32.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:38:32.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:38:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:33.300+0000 7fb9d4815640 -1 osd.0 15 log_to_monitors true 2024-09-10T21:38:33.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:38:33.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:38:33.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:38:33.750 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:38:33.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:38:33.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:38:34.057 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 16 last_clean_interval [5,15) [v2:127.0.0.1:6810/1802062164,v1:127.0.0.1:6811/1802062164] [v2:127.0.0.1:6812/1802062164,v1:127.0.0.1:6813/1802062164] exists,up 2bdd5acf-4c77-49d8-9393-5460f86a05e4 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:38:34.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:38:34.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:38:34.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:38:34.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:38:34.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:38:34.427 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:38:34.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:38:34.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:34.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:38:34.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280514 2024-09-10T21:38:34.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280514 2024-09-10T21:38:34.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280514' 2024-09-10T21:38:34.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:34.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:38:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378626 2024-09-10T21:38:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378626 2024-09-10T21:38:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280514 1-81604378626' 2024-09-10T21:38:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:34.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280514 2024-09-10T21:38:34.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:34.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:38:34.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280514 2024-09-10T21:38:34.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:34.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280514 2024-09-10T21:38:34.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2024-09-10T21:38:34.662 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 94489280514 2024-09-10T21:38:34.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:34.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 94489280514 2024-09-10T21:38:34.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:38:35.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:38:35.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 94489280514 2024-09-10T21:38:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:38:37.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:38:37.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:37.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280514 -lt 94489280514 2024-09-10T21:38:37.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:37.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-81604378626 2024-09-10T21:38:37.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:37.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:38:37.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-81604378626 2024-09-10T21:38:37.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:37.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378626 2024-09-10T21:38:37.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 81604378626' 2024-09-10T21:38:37.519 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 81604378626 2024-09-10T21:38:37.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:38:37.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378627 -lt 81604378626 2024-09-10T21:38:37.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:38:37.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:37.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:38:38.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:38:38.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T21:38:38.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:38:38.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:38.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:38.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T21:38:38.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T21:38:38.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T21:38:38.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T21:38:38.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:38:38.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T21:38:38.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:38:38.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:38:38.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:38:38.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:38:38.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:38:38.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:38:38.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:38:38.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:38:39.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T21:38:39.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:38:39.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:39.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:38:39.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:38:39.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:38:39.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:38:39.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:38:39.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:38:39.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:38:39.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T21:38:39.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:38:39.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T21:38:40.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:38:40.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:38:40.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:38:40.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T21:38:40.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:38:40.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:40.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T21:38:40.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T21:38:40.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T21:38:40.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T21:38:40.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-10T21:38:40.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T21:38:40.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T21:38:41.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T21:38:41.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:38:41.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:38:41.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:38:41.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:38:41.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:38:41.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:38:41.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:38:41.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:38:41.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:38:41.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:41.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:38:42.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:38:42.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:38:42.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:42.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T21:38:42.541 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:38:42.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:38:42.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:38:42.554 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:38:42.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:37:43.695316+0000 '>' 2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:42.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:38:43.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:38:43.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:38:43.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:38:43.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:38:43.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:38:43.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:38:43.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:38:44.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:37:43.695316+0000 '>' 2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:44.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:38:45.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:38:45.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:38:45.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:38:45.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:38:45.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:38:45.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:38:45.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:38:45.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:37:43.695316+0000 '>' 2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:45.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:38:46.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:38:46.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:38:46.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:38:46.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:38:46.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:38:46.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:38:46.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:38:46.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:38:42.734728+0000 '>' 2024-09-10T21:37:43.695316+0000 2024-09-10T21:38:46.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:38:46.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:38:48.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-09-10T21:38:48.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:38:48.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:38:48.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:38:48.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:38:48.703 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:38:48.703 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:38:48.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:38:48.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:38:48.704 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:38:48.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:38:49.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:38:49.087 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:38:49.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:38:49.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:49.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:38:49.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280518 2024-09-10T21:38:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280518 2024-09-10T21:38:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280518' 2024-09-10T21:38:49.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:49.191 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378630 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378630 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280518 1-81604378630' 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280518 2024-09-10T21:38:49.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:49.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:38:49.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280518 2024-09-10T21:38:49.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:49.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280518 2024-09-10T21:38:49.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280518' 2024-09-10T21:38:49.296 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 94489280518 2024-09-10T21:38:49.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280516 -lt 94489280518 2024-09-10T21:38:49.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:38:50.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:38:50.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:50.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280518 -lt 94489280518 2024-09-10T21:38:50.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:50.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-81604378630 2024-09-10T21:38:50.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:50.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:38:50.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-81604378630 2024-09-10T21:38:50.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:50.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378630 2024-09-10T21:38:50.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 81604378630' 2024-09-10T21:38:50.868 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 81604378630 2024-09-10T21:38:50.868 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:38:51.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378631 -lt 81604378630 2024-09-10T21:38:51.143 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:38:51.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:51.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:51.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:38:51.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:38:51.488 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:38:51.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:38:51.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:38:51.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:38:51.763 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:38:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:38:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:38:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:38:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-09-10T21:38:52.364 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T21:38:52.364 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-10T21:38:43.415694+0000 15'30 23:122 [1,0] 1 [1,0] 1 15'30 2024-09-10T21:38:42.734728+0000 15'30 2024-09-10T21:38:42.734728+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:38:42.734728+0000 20 0 2024-09-10T21:38:52.365 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T21:38:52.365 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T21:38:52.365 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:38:52.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-09-10T21:38:52.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:38:52.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:38:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:38:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:38:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:38:52.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:52.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:38:52.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=94489280519 2024-09-10T21:38:52.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 94489280519 2024-09-10T21:38:52.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280519' 2024-09-10T21:38:52.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:38:52.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:38:52.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378632 2024-09-10T21:38:52.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378632 2024-09-10T21:38:52.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-94489280519 1-81604378632' 2024-09-10T21:38:52.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:52.879 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-94489280519 2024-09-10T21:38:52.879 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:52.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:38:52.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-94489280519 2024-09-10T21:38:52.880 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:52.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=94489280519 2024-09-10T21:38:52.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2024-09-10T21:38:52.882 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 94489280519 2024-09-10T21:38:52.882 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280518 -lt 94489280519 2024-09-10T21:38:53.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:38:54.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:38:54.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:54.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280518 -lt 94489280519 2024-09-10T21:38:54.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:38:55.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:38:55.442 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:38:55.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 94489280520 -lt 94489280519 2024-09-10T21:38:55.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:38:55.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-81604378632 2024-09-10T21:38:55.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:38:55.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:38:55.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-81604378632 2024-09-10T21:38:55.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:38:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378632 2024-09-10T21:38:55.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 81604378632' 2024-09-10T21:38:55.732 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 81604378632 2024-09-10T21:38:55.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:38:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378632 -lt 81604378632 2024-09-10T21:38:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-09-10T21:38:56.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes": 210, 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects": 30, 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_object_clones": 0, 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_object_copies": 60, 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T21:38:56.113 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_missing": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_degraded": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_misplaced": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_unfound": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_dirty": 30, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_whiteouts": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_read": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_read_kb": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_write": 30, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_write_kb": 30, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrub_errors": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_deep_scrub_errors": 0, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_recovered": 20, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes_recovered": 140, 2024-09-10T21:38:56.114 INFO:tasks.workunit.client.0.smithi181.stdout: "num_keys_recovered": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_omap": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_kb": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_kb": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_promote": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_mode_high": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_mode_low": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_mode_some": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_mode_full": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_pinned": 0, 2024-09-10T21:38:56.115 INFO:tasks.workunit.client.0.smithi181.stdout: "num_legacy_snapsets": 0, 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_large_omap_objects": 0, 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_manifest": 0, 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_omap_bytes": 0, 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_omap_keys": 0, 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_repaired": 20 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:38:56.116 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-09-10T21:38:56.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-09-10T21:38:56.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-09-10T21:38:56.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-09-10T21:38:56.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:56.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-10T21:38:56.475 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 19, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 81604378632, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 1, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 1, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 1, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 1, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 104857600, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 28552, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 312, 2024-09-10T21:38:56.486 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 4, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 28219, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 104829048, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 107374182400, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 107344945152, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 319488, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 51326, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 4397, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 28896979 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:56.487 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [ 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: 0 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 10, 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:56.488 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [] 2024-09-10T21:38:56.489 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:38:56.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:56.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-10T21:38:56.756 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:56.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-09-10T21:38:56.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-09-10T21:38:56.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-09-10T21:38:56.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:56.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-10T21:38:57.025 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:57.036 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 22, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 94489280520, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 1, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 1, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 1, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 1, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 104857600, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 28552, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 312, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 4, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 28219, 2024-09-10T21:38:57.037 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 104829048, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 107374182400, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 107344945152, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 319488, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 51326, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:38:57.038 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 4401, 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 28896975 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [ 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 10, 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:38:57.039 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [] 2024-09-10T21:38:57.040 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:38:57.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:57.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-10T21:38:57.293 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:57.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-09-10T21:38:57.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-09-10T21:38:57.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-09-10T21:38:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:57.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-09-10T21:38:57.562 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:57.573 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:38:57.573 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 0, 2024-09-10T21:38:57.573 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:38:57.573 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 2, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 2, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 2, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 2, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 209715200, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 57104, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 624, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 8, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 56439, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 209658096, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 214748364800, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 214689890304, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 638976, 2024-09-10T21:38:57.574 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 102652, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 8798, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 57793954 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [], 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 20, 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:38:57.575 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [], 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout: "network_ping_times": [] 2024-09-10T21:38:57.576 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:38:57.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-10T21:38:57.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-10T21:38:57.834 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:38:57.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:38:57.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:38:57.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:38:57.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:38:57.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:38:57.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:38:57.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:38:57.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:38:57.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:38:57.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:38:57.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:38:57.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:38:57.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:38:57.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:38:57.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:38:57.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:38:57.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:38:57.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:38:57.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:38:57.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:38:57.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:38:57.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:57.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:57.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:38:57.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:38:57.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:38:57.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:38:57.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:38:57.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:38:57.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:38:57.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:38:57.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:38:57.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:38:57.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:38:57.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:38:57.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:38:57.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:38:57.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:38:57.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:38:57.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:38:57.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:38:57.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:38:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:38:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:38:57.985 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:38:57.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:38:57.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:38:57.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:57.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:57.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:38:57.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:38:57.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:38:57.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:38:57.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:38:57.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:38:57.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:38:57.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:38:58.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:58.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:38:58.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:38:58.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:38:58.629 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:38:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:38:58.630 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:58.631 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:38:58.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:38:58.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:38:58.714 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:38:58.715 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:38:58.715 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:38:58.715 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:58.716 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:58.716 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:38:58.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:38:58.716 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:38:58.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:38:59.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:38:59.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:59.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:38:59.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:38:59.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:38:59.635 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-09-10T21:38:59.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-09-10T21:38:59.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:38:59.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:38:59.642 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:38:59.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:38:59.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:38:59.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:38:59.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=392c8457-144e-4a93-87d0-b6624c93bc30 2024-09-10T21:38:59.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 392c8457-144e-4a93-87d0-b6624c93bc30' 2024-09-10T21:38:59.644 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 392c8457-144e-4a93-87d0-b6624c93bc30 2024-09-10T21:38:59.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:38:59.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBzvOBms59MJxAAiac9Enun9tMWWhFza0q4Lw== 2024-09-10T21:38:59.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBzvOBms59MJxAAiac9Enun9tMWWhFza0q4Lw=="}' 2024-09-10T21:38:59.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 392c8457-144e-4a93-87d0-b6624c93bc30 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:38:59.862 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:38:59.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:38:59.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBzvOBms59MJxAAiac9Enun9tMWWhFza0q4Lw== --osd-uuid 392c8457-144e-4a93-87d0-b6624c93bc30 2024-09-10T21:38:59.903 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:59.902+0000 7fb6e9818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:59.911+0000 7fb6e9818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:59.913 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:59.912+0000 7fb6e9818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:38:59.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:59.913+0000 7fb6e9818640 -1 bdev(0x56498e75fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:38:59.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:38:59.913+0000 7fb6e9818640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:39:05.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:39:05.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:39:05.345 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:39:05.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:39:05.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:39:05.711 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:39:05.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:39:05.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:39:05.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:39:05.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:39:05.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:39:05.740 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:05.738+0000 7f63c601f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:05.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:05.740+0000 7f63c601f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:05.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:05.742+0000 7f63c601f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:05.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:39:05.999 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:39:05.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:05.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:06.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:06.858+0000 7f63c601f640 -1 Falling back to public interface 2024-09-10T21:39:07.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:07.276 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:39:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:39:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:07.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:07.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:08.460 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:08.459+0000 7f63c601f640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:39:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:08.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:08.562 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:39:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:39:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:08.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:08.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:09.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:09.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:09.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:39:09.917 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:39:09.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:09.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:10.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:11.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:11.497 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:12.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:12.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:12.499 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:39:12.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:39:12.499 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:12.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:12.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:13.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:13.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:13.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:39:13.788 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:39:13.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:13.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:14.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:15.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:15.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:15.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:39:15.078 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:39:15.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:15.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:15.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:16.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:16.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:16.428 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:39:16.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:39:16.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:16.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:16.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:17.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:17.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:17.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:39:17.748 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:39:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:17.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:18.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:39:19.043 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:39:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:19.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:19.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:20.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:39:20.432 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:39:20.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:20.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:20.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:21.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:39:21.924 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:39:21.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:21.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:22.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:23.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:23.731 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:39:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:39:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:23.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:24.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:25.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:25.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:39:25.378 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:39:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:27.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:27.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:27.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:39:27.172 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:39:27.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:27.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:27.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:28.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:28.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:28.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:39:28.591 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:39:28.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:28.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:29.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:30.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:30.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:30.016 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:39:30.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:39:30.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:30.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:30.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:31.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:39:31.778 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:39:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:31.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:39:32.063 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/26491682,v1:127.0.0.1:6803/26491682] [v2:127.0.0.1:6804/26491682,v1:127.0.0.1:6805/26491682] exists,up 392c8457-144e-4a93-87d0-b6624c93bc30 2024-09-10T21:39:32.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:39:32.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:39:32.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:39:32.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:39:32.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:39:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db9fa291-422c-4c0a-80b4-3db24a3335c3 2024-09-10T21:39:32.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 db9fa291-422c-4c0a-80b4-3db24a3335c3' 2024-09-10T21:39:32.070 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 db9fa291-422c-4c0a-80b4-3db24a3335c3 2024-09-10T21:39:32.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:39:32.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUvOBm+3AoBRAADMzMX4VlozQJcO+s+1I/jg== 2024-09-10T21:39:32.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUvOBm+3AoBRAADMzMX4VlozQJcO+s+1I/jg=="}' 2024-09-10T21:39:32.088 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db9fa291-422c-4c0a-80b4-3db24a3335c3 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:39:32.445 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:39:32.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:39:32.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCUvOBm+3AoBRAADMzMX4VlozQJcO+s+1I/jg== --osd-uuid db9fa291-422c-4c0a-80b4-3db24a3335c3 2024-09-10T21:39:32.488 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:32.487+0000 7f476c011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:32.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:32.529+0000 7f476c011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:32.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:32.530+0000 7f476c011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:32.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:32.530+0000 7f476c011640 -1 bdev(0x56073ce45c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:39:32.531 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:32.530+0000 7f476c011640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:39:38.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:39:38.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:39:38.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:39:38.836 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:39:38.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:39:39.201 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:39:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:39:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:39:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:39:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:39:39.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:39:39.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:39.229+0000 7f6002e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:39.232 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:39.231+0000 7f6002e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:39.233 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:39.232+0000 7f6002e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:39:39.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:39:39.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:39:39.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:39:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:39:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:39:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:39.478 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:39:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:39:39.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:39.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:39.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:40.602 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:40.601+0000 7f6002e30640 -1 Falling back to public interface 2024-09-10T21:39:40.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:40.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:40.764 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:39:40.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:39:40.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:40.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:42.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:42.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:42.050 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:39:42.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:39:42.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:42.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:42.260 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:39:42.258+0000 7f6002e30640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:39:42.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:43.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:43.361 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:39:43.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:43.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:39:43.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:43.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:43.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:44.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:44.647 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:39:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:39:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:44.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:45.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:45.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:45.941 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:39:45.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:39:45.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:45.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:47.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:47.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:47.252 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:39:47.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:39:47.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:47.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:47.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:48.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:48.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:48.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:39:48.540 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:39:48.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:48.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:48.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:49.868 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:39:49.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:39:49.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:49.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:50.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:51.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:51.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:51.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:39:51.237 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:39:51.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:51.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:51.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:52.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:52.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:52.524 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:39:52.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:39:52.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:52.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:52.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:53.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:53.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:53.812 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:39:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:39:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:53.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:54.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:55.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:55.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:55.296 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:39:55.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:39:55.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:55.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:55.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:56.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:56.773 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:39:56.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:56.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:39:56.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:56.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:57.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:58.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:58.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:58.450 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:39:58.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:39:58.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:58.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:39:58.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:39:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:39:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:39:59.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:39:59.739 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:39:59.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:39:59.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:40:00.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:01.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:01.363 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:40:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:40:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:01.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:40:02.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:03.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:03.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:03.034 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:40:03.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:40:03.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:03.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:40:03.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:04.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:04.640 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:40:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:40:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:04.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:40:05.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:06.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:06.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:06.178 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:40:06.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:40:06.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:06.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:40:06.472 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1454687305,v1:127.0.0.1:6811/1454687305] [v2:127.0.0.1:6812/1454687305,v1:127.0.0.1:6813/1454687305] exists,up db9fa291-422c-4c0a-80b4-3db24a3335c3 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:40:06.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:40:06.476 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:40:06.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:40:06.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:40:06.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:40:06.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6f92e55e-c514-464d-83f9-4ef0e6273c65 2024-09-10T21:40:06.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6f92e55e-c514-464d-83f9-4ef0e6273c65' 2024-09-10T21:40:06.479 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 6f92e55e-c514-464d-83f9-4ef0e6273c65 2024-09-10T21:40:06.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:40:06.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2vOBmDsV2HRAAh+UffE8OLAqIR0vgWfjqpw== 2024-09-10T21:40:06.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2vOBmDsV2HRAAh+UffE8OLAqIR0vgWfjqpw=="}' 2024-09-10T21:40:06.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6f92e55e-c514-464d-83f9-4ef0e6273c65 -i td/osd-scrub-repair/2/new.json 2024-09-10T21:40:06.871 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:40:06.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:40:06.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC2vOBmDsV2HRAAh+UffE8OLAqIR0vgWfjqpw== --osd-uuid 6f92e55e-c514-464d-83f9-4ef0e6273c65 2024-09-10T21:40:06.914 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:06.912+0000 7fdac541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:06.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:06.962+0000 7fdac541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:06.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:06.963+0000 7fdac541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:06.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:06.963+0000 7fdac541c640 -1 bdev(0x557c082cbc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:40:06.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:06.963+0000 7fdac541c640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:40:13.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:40:13.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:40:13.146 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:40:13.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:40:13.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:40:13.512 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:40:13.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:40:13.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:40:13.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:40:13.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:40:13.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:40:13.541 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:13.540+0000 7f70e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:13.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:13.542+0000 7f70e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:13.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:13.543+0000 7f70e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:40:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:13.817 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:40:13.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:40:13.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:13.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:14.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:14.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:14.408+0000 7f70e6c1a640 -1 Falling back to public interface 2024-09-10T21:40:15.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:15.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:15.097 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:40:15.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:40:15.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:15.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:15.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:16.270 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:40:16.269+0000 7f70e6c1a640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:40:16.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:16.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:16.384 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:40:16.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:40:16.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:16.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:16.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:17.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:17.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:17.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:40:17.782 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:40:17.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:17.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:18.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:19.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:19.095 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:40:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:40:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:19.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:19.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:20.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:20.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:20.383 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:40:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:40:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:20.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:20.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:21.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:21.670 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:40:21.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:40:21.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:21.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:22.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:23.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:23.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:23.046 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:40:23.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:40:23.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:23.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:23.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:24.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:24.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:24.331 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:40:24.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:40:24.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:24.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:24.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:25.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:25.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:25.703 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:40:25.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:40:25.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:25.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:26.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:27.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:27.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:27.067 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:40:27.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:40:27.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:27.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:27.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:28.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:28.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:28.353 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:40:28.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:40:28.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:28.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:28.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:29.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:29.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:29.662 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:40:29.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:40:29.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:29.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:30.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:31.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:31.422 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:40:31.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:40:31.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:31.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:32.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:33.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:33.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:33.143 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:40:33.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:40:33.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:33.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:33.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:34.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:34.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:34.759 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:40:34.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:40:34.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:34.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:35.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:36.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:36.626 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:40:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:40:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:36.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:37.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:38.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:38.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:38.247 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:40:38.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:40:38.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:38.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:38.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:40:39.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:40:39.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:40:39.660 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:40:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:40:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:40:39.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:40:39.951 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2014116775,v1:127.0.0.1:6819/2014116775] [v2:127.0.0.1:6820/2014116775,v1:127.0.0.1:6821/2014116775] exists,up 6f92e55e-c514-464d-83f9-4ef0e6273c65 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=testpool 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T21:40:39.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:40:39.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T21:40:40.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-09-10T21:40:40.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-09-10T21:40:40.972 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'testpool' created 2024-09-10T21:40:40.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:40:41.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T21:40:41.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:40:41.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:40:41.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:40:41.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:40:41.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:40:41.988 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:40:41.988 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:40:41.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:40:41.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:40:41.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:40:42.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:40:42.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:40:42.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:40:42.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:40:42.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:40:42.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:40:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:40:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:40:42.370 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:40:42.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:40:42.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:40:42.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:40:42.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-10T21:40:42.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-10T21:40:42.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-10T21:40:42.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:40:42.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:40:42.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-10T21:40:42.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-10T21:40:42.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-09-10T21:40:42.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:40:42.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:40:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-10T21:40:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-10T21:40:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672971 2-64424509446' 2024-09-10T21:40:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:40:42.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-10T21:40:42.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:40:42.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:40:42.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-10T21:40:42.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:40:42.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-10T21:40:42.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-10T21:40:42.738 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836500 2024-09-10T21:40:42.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:40:43.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-10T21:40:43.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:40:44.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:40:44.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:40:44.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-10T21:40:44.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:40:44.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-10T21:40:44.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:40:44.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:40:44.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-10T21:40:44.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:40:44.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-10T21:40:44.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-10T21:40:44.305 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672971 2024-09-10T21:40:44.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:40:44.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-10T21:40:44.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:40:44.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-10T21:40:44.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:40:44.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:40:44.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-10T21:40:44.583 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:40:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-10T21:40:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-10T21:40:44.585 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509446 2024-09-10T21:40:44.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:40:44.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-10T21:40:44.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:40:44.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:40:44.859 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:40:45.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:40:45.206 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:40:45.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:40:45.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:40:45.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:40:45.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:40:45.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-09-10T21:40:45.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:45.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:45.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:45.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:46.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:46.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:46.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:46.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:46.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:46.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:46.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:46.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:47.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:47.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:47.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:47.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:47.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:47.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:47.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:47.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:47.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:47.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:48.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:48.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:48.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:48.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:48.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:48.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:48.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:48.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:48.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:49.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:49.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:49.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:49.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:49.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:49.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:49.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:49.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:50.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:50.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:50.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:50.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:50.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:50.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:50.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:50.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:50.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:51.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:51.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:51.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:51.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:51.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:51.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:51.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:51.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:52.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-10T21:40:52.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T21:40:52.247 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:40:52.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:40:52.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-10T21:40:52.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:40:52.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-09-10T21:40:52.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-09-10T21:40:52.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-10T21:40:52.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-10T21:40:52.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-10T21:40:52.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:40:53.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-09-10T21:40:53.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-09-10T21:40:53.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-10T21:40:53.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:40:53.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-10T21:40:53.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:40:53.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-09-10T21:40:53.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:40:53.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:40:53.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:40:53.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:40:53.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:40:53.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:40:53.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:40:53.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-10T21:40:53.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:40:53.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:40:53.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:40:53.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:40:53.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:40:54.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:40:55.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:40:55.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-10T21:40:56.483 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-09-10T21:40:56.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:40:56.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:40:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:40:56.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:40:56.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-10T21:40:58.526 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:104778fc:::obj2:head# 2024-09-10T21:40:59.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:40:59.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-09-10T21:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-10T21:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:40:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:40:59.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-10T21:41:00.224 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-09-10T21:41:00.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:00.742 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-09-10T21:41:00.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:00.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:00.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-10T21:41:01.868 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-09-10T21:41:02.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:02.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-09-10T21:41:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-10T21:41:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:02.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:02.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-10T21:41:03.329 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:c52c9666:::obj5:head# 2024-09-10T21:41:03.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:03.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-09-10T21:41:03.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:03.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-10T21:41:03.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:03.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:03.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:03.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:03.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:03.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-10T21:41:04.948 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:233a42c1:::obj6:head# 2024-09-10T21:41:05.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:05.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:05.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-10T21:41:06.409 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:9a09113a:::obj7:head# 2024-09-10T21:41:06.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:06.924 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-09-10T21:41:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-10T21:41:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-10T21:41:07.994 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:a557efb1:::obj8:head# 2024-09-10T21:41:08.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:08.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:08.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:08.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-10T21:41:09.454 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:fe351e27:::obj9:head# 2024-09-10T21:41:09.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:09.973 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-09-10T21:41:09.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:09.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:09.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-10T21:41:10.898 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:d1337354:::obj10:head# 2024-09-10T21:41:11.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:11.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:11.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:11.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:11.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-10T21:41:12.342 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-09-10T21:41:13.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:13.121 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:13.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:13.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-10T21:41:14.146 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-09-10T21:41:14.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:14.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:14.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-10T21:41:15.924 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-09-10T21:41:16.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:16.439 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-10T21:41:17.343 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:e0a44829:::obj14:head# 2024-09-10T21:41:17.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:17.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-09-10T21:41:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-10T21:41:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:17.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:17.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:17.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:17.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-10T21:41:18.719 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:ab946124:::obj15:head# 2024-09-10T21:41:19.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:19.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-09-10T21:41:19.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:19.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-10T21:41:20.172 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-09-10T21:41:20.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:20.687 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-09-10T21:41:20.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:20.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-10T21:41:20.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:20.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-10T21:41:21.598 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-09-10T21:41:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:22.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:22.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:22.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-10T21:41:23.034 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:b9836a99:::obj18:head# 2024-09-10T21:41:23.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:23.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:23.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-10T21:41:24.444 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-09-10T21:41:24.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:24.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-09-10T21:41:24.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:24.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-10T21:41:24.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:24.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:24.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:24.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:24.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:24.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-10T21:41:26.057 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:24b5611b:::obj20:head# 2024-09-10T21:41:26.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:26.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-09-10T21:41:26.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:26.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-09-10T21:41:26.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:26.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:26.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:26.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:26.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:26.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-09-10T21:41:27.517 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-09-10T21:41:28.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:28.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-09-10T21:41:28.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:28.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-09-10T21:41:28.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:28.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:28.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:28.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:28.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:28.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-09-10T21:41:28.986 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:c0c49def:::obj22:head# 2024-09-10T21:41:29.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:29.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-09-10T21:41:29.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-09-10T21:41:30.463 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:242c40e0:::obj23:head# 2024-09-10T21:41:30.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:30.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-09-10T21:41:30.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:30.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-09-10T21:41:30.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:30.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:30.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:30.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-09-10T21:41:32.083 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:7eacede5:::obj24:head# 2024-09-10T21:41:32.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:32.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T21:41:32.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:32.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-09-10T21:41:33.551 INFO:tasks.workunit.client.0.smithi181.stdout:remove 0#1:7109478f:::obj25:head# 2024-09-10T21:41:34.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-10T21:41:34.068 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-09-10T21:41:34.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-10T21:41:34.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-09-10T21:41:34.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:41:34.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:41:34.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:41:34.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:41:34.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:34.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-09-10T21:41:35.011 INFO:tasks.workunit.client.0.smithi181.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:41:35.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:41:35.530 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:41:35.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:41:35.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:41:35.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:41:35.532 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:41:35.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:41:35.533 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-10T21:41:35.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:41:35.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:41:35.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:41:35.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:41:35.563 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:35.563+0000 7f0153a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:35.566 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:35.565+0000 7f0153a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:35.567 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:35.566+0000 7f0153a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:35.842 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:41:35.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:41:35.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:35.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:41:36.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:37.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:37.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:37.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:41:37.122 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:41:37.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:37.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:41:37.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:37.398+0000 7f0153a0b640 -1 Falling back to public interface 2024-09-10T21:41:37.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:38.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:38.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:38.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:41:38.412 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:41:38.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:38.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:41:38.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:38.983 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:38.982+0000 7f0153a0b640 -1 osd.1 22 log_to_monitors true 2024-09-10T21:41:39.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:39.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:39.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:41:39.701 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:41:39.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:39.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:41:40.026 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/4289426295,v1:127.0.0.1:6803/4289426295] [v2:127.0.0.1:6804/4289426295,v1:127.0.0.1:6805/4289426295] exists,up db9fa291-422c-4c0a-80b4-3db24a3335c3 2024-09-10T21:41:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:41:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:41:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:41:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:41:40.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-10T21:41:40.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:41:40.033 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:41:40.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:41:40.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:41:40.058 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:40.057+0000 7fe4c6ada640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:40.060 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:40.059+0000 7fe4c6ada640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:40.061 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:40.060+0000 7fe4c6ada640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:40.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:41:40.350 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:41:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:40.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:41:40.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:41.209 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:41.208+0000 7fe4c6ada640 -1 Falling back to public interface 2024-09-10T21:41:41.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:41.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:41.627 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:41:41.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:41:41.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:41.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:41:41.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:42.729 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:41:42.727+0000 7fe4c6ada640 -1 osd.0 21 log_to_monitors true 2024-09-10T21:41:42.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:42.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:42.915 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:41:42.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:41:42.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:42.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:41:43.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:41:44.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:41:44.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:41:44.213 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:41:44.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:41:44.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:41:44.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:41:44.522 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/881892459,v1:127.0.0.1:6811/881892459] [v2:127.0.0.1:6812/881892459,v1:127.0.0.1:6813/881892459] exists,up 392c8457-144e-4a93-87d0-b6624c93bc30 2024-09-10T21:41:44.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:41:44.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:41:44.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:41:44.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:41:44.893 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:41:44.893 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:41:44.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:41:44.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:44.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:41:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-10T21:41:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-10T21:41:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-10T21:41:44.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:44.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:41:45.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-10T21:41:45.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-10T21:41:45.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-09-10T21:41:45.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:45.100 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:41:45.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-10T21:41:45.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-10T21:41:45.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509460' 2024-09-10T21:41:45.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:41:45.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-10T21:41:45.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:41:45.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:41:45.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-10T21:41:45.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:41:45.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-10T21:41:45.205 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 124554051586 2024-09-10T21:41:45.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-10T21:41:45.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:41:45.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-10T21:41:45.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:41:46.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:41:46.480 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:41:46.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-10T21:41:46.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:41:46.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149699 2024-09-10T21:41:46.767 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:41:46.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:41:46.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149699 2024-09-10T21:41:46.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:41:46.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-10T21:41:46.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-09-10T21:41:46.770 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 111669149699 2024-09-10T21:41:46.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:41:47.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-10T21:41:47.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:41:47.047 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-10T21:41:47.048 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:41:47.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:41:47.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-10T21:41:47.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:41:47.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-10T21:41:47.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-10T21:41:47.051 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509460 2024-09-10T21:41:47.051 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:41:47.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-10T21:41:47.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:41:47.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:41:47.326 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:41:47.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:41:47.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:41:47.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:41:47.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:41:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:41:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:41:47.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T21:41:48.313 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:48.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:48.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:48.314 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:48.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:48.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T21:41:48.841 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:41:48.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:48.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:48.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:48.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:48.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:48.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:49.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:49.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:50.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:50.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:50.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:50.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:50.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:50.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:50.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:50.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:50.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:51.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:51.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:51.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:51.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:51.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:52.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:52.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:52.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:52.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:52.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:52.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:52.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:53.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:53.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:54.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:54.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:54.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:54.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:54.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:54.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:54.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:54.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:54.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:55.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:55.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:55.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:55.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:55.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:55.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:55.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:55.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:40:40.862639+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:55.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:41:56.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:41:56.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:41:56.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:41:56.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:41:56.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:41:56.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:41:56.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:41:56.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:41:49.897369+0000 '>' 2024-09-10T21:40:40.862639+0000 2024-09-10T21:41:56.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:41:56.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:41:58.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-09-10T21:41:58.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:41:58.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:41:58.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:41:58.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:41:58.862 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:41:58.862 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:41:58.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:41:58.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:41:58.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:41:58.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:41:58.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:41:58.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:41:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:41:58.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:41:58.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:41:59.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:41:59.244 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:41:59.244 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:41:59.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:41:59.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:59.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:41:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051590 2024-09-10T21:41:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051590 2024-09-10T21:41:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051590' 2024-09-10T21:41:59.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:59.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:41:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149703 2024-09-10T21:41:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149703 2024-09-10T21:41:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051590 1-111669149703' 2024-09-10T21:41:59.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:41:59.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:41:59.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-10T21:41:59.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-10T21:41:59.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051590 1-111669149703 2-64424509463' 2024-09-10T21:41:59.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:41:59.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051590 2024-09-10T21:41:59.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:41:59.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:41:59.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051590 2024-09-10T21:41:59.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:41:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051590 2024-09-10T21:41:59.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051590' 2024-09-10T21:41:59.574 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 124554051590 2024-09-10T21:41:59.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:41:59.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051590 2024-09-10T21:41:59.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:41:59.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149703 2024-09-10T21:41:59.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:41:59.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:41:59.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149703 2024-09-10T21:41:59.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:41:59.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149703 2024-09-10T21:41:59.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149703' 2024-09-10T21:41:59.854 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 111669149703 2024-09-10T21:41:59.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:42:00.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149703 -lt 111669149703 2024-09-10T21:42:00.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:42:00.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-10T21:42:00.174 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:42:00.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:42:00.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-10T21:42:00.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:42:00.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-10T21:42:00.177 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509463 2024-09-10T21:42:00.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-10T21:42:00.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:42:00.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-10T21:42:00.452 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:42:00.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:42:00.453 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:42:00.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:42:00.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:42:00.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:42:00.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:42:00.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:42:00.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:42:00.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:42:00.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:42:01.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:42:01.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:42:01.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:42:01.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:42:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:42:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:42:01.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:42:01.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-09-10T21:42:01.678 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T21:42:01.678 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-10T21:41:51.026104+0000 21'30 30:132 [1,0,2] 1 [1,0,2] 1 21'30 2024-09-10T21:41:49.897369+0000 21'30 2024-09-10T21:41:49.897369+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:41:49.897369+0000 17 0 2024-09-10T21:42:01.678 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T21:42:01.678 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T21:42:01.678 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:42:01.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-09-10T21:42:01.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:42:01.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:42:01.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:42:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:42:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T21:42:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:42:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:42:01.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:42:02.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051591 2024-09-10T21:42:02.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051591 2024-09-10T21:42:02.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591' 2024-09-10T21:42:02.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:42:02.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:42:02.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149704 2024-09-10T21:42:02.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149704 2024-09-10T21:42:02.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-09-10T21:42:02.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:42:02.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:42:02.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-10T21:42:02.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-10T21:42:02.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509465' 2024-09-10T21:42:02.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:42:02.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051591 2024-09-10T21:42:02.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:42:02.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:42:02.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051591 2024-09-10T21:42:02.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:42:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051591 2024-09-10T21:42:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-09-10T21:42:02.281 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 124554051591 2024-09-10T21:42:02.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:42:02.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051591 2024-09-10T21:42:02.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:42:03.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:42:03.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:42:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051592 -lt 124554051591 2024-09-10T21:42:03.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:42:03.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149704 2024-09-10T21:42:03.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:42:03.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:42:03.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149704 2024-09-10T21:42:03.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:42:03.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149704 2024-09-10T21:42:03.851 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 111669149704 2024-09-10T21:42:03.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-09-10T21:42:03.851 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:42:04.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149704 -lt 111669149704 2024-09-10T21:42:04.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:42:04.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-10T21:42:04.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:42:04.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:42:04.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-10T21:42:04.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:42:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-10T21:42:04.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-10T21:42:04.132 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509465 2024-09-10T21:42:04.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:42:04.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-10T21:42:04.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-10T21:42:04.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes": 210, 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects": 30, 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout: "num_object_clones": 0, 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout: "num_object_copies": 90, 2024-09-10T21:42:04.514 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_missing": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_degraded": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_misplaced": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_unfound": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_dirty": 30, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_whiteouts": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_read": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_read_kb": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_write": 30, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_write_kb": 30, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrub_errors": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_deep_scrub_errors": 0, 2024-09-10T21:42:04.515 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_recovered": 26, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes_recovered": 182, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_keys_recovered": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_omap": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_kb": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_kb": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_promote": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_mode_high": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_flush_mode_low": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_mode_some": 0, 2024-09-10T21:42:04.516 INFO:tasks.workunit.client.0.smithi181.stdout: "num_evict_mode_full": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_pinned": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_legacy_snapsets": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_large_omap_objects": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_manifest": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_omap_bytes": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_omap_keys": 0, 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout: "num_objects_repaired": 26 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:04.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-10T21:42:04.518 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-09-10T21:42:04.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-09-10T21:42:04.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-09-10T21:42:04.621 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-09-10T21:42:04.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-09-10T21:42:04.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:42:04.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-09-10T21:42:04.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-10T21:42:04.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-10T21:42:04.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:04.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-10T21:42:04.884 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:04.895 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:04.895 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:42:04.895 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 29, 2024-09-10T21:42:04.895 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 124554051592, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 1, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 1, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 1, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 1, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 104857600, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 28480, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 368, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 4, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 28091, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 104829120, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 107374182400, 2024-09-10T21:42:04.896 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 107345018880, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 376832, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 203264, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 4402, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 28765902 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [ 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: 1, 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: 2 2024-09-10T21:42:04.897 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 13, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:42:04.898 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:42:04.899 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:04.899 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [] 2024-09-10T21:42:04.899 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:04.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:04.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-10T21:42:05.153 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:05.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-10T21:42:05.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-10T21:42:05.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:42:05.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-09-10T21:42:05.165 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-10T21:42:05.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-10T21:42:05.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:05.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-10T21:42:05.425 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:05.436 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:05.436 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 26, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 111669149704, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 1, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 1, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 1, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 1, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 104857600, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 28480, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 368, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 4, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 28091, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 104829120, 2024-09-10T21:42:05.437 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 107374182400, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 107345018880, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 376832, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 203264, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 4398, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 28765906 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [ 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-10T21:42:05.438 INFO:tasks.workunit.client.0.smithi181.stdout: 2 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 13, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.439 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [] 2024-09-10T21:42:05.440 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:05.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:05.440 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-10T21:42:05.697 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:05.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:05.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-09-10T21:42:05.967 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 2, 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 15, 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 64424509466, 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 1, 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 1, 2024-09-10T21:42:05.978 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 1, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 1, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 104857600, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 27212, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 368, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 1, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 26814, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 104830388, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 107374182400, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 107346317312, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 376832, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 203264, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:42:05.979 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 1594, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 27457990 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [ 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 0, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.980 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [] 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:05.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-09-10T21:42:06.238 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:06.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-09-10T21:42:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-09-10T21:42:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:06.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-09-10T21:42:06.509 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:06.520 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "up_from": 0, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "num_pgs": 3, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "num_osds": 3, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_osds": 3, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "kb": 314572800, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used": 84172, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_data": 1104, 2024-09-10T21:42:06.521 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_omap": 10, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_used_meta": 82997, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "kb_avail": 314488628, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "statfs": { 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "total": 322122547200, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "available": 322036355072, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "internally_reserved": 0, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "allocated": 1130496, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "data_stored": 609792, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed": 0, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_allocated": 0, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "data_compressed_original": 0, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_allocated": 10394, 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: "internal_metadata": 84989798 2024-09-10T21:42:06.522 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "hb_peers": [], 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "snap_trim_queue_len": 0, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "num_snap_trimming": 0, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "num_shards_repaired": 26, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "op_queue_age_hist": { 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "histogram": [], 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "upper_bound": 1 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "perf_stat": { 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ms": 0, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ms": 0, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "commit_latency_ns": 0, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "apply_latency_ns": 0 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:42:06.523 INFO:tasks.workunit.client.0.smithi181.stdout: "alerts": [], 2024-09-10T21:42:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "network_ping_times": [] 2024-09-10T21:42:06.524 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:06.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-10T21:42:06.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-10T21:42:06.780 INFO:tasks.workunit.client.0.smithi181.stderr:dumped all 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:42:06.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:42:06.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:42:06.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:42:06.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:42:06.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:42:06.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:42:06.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:42:06.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:42:06.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:42:06.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:42:06.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:42:06.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:42:06.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:42:06.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:42:06.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:42:06.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:42:06.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:42:06.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:42:06.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:06.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:06.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:42:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:42:06.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:42:06.928 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:42:06.929 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:42:06.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:42:06.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:42:06.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:42:06.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:42:06.932 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:42:06.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:42:06.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:42:06.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:42:06.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:42:06.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:42:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:42:06.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:42:06.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:42:06.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:06.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:06.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:42:06.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:42:06.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:42:06.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:42:06.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:42:06.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:06.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:06.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local poolname=psr_pool 2024-09-10T21:42:06.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6202: TEST_request_scrub_priority: local objname=POBJ 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: local OBJECTS=64 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: local PGS=8 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:42:06.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:42:07.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:42:07.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:42:07.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:42:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:42:07.424 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:07.425 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:07.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:42:07.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:42:07.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:42:07.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:42:07.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:42:07.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:42:07.600 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:42:07.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:42:07.601 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:42:07.601 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:42:07.602 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:42:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:42:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:42:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:42:07.680 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:42:07.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:42:07.681 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:42:07.681 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:42:07.681 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:42:07.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:42:07.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:42:07.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:42:08.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:42:08.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:42:08.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:42:08.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6208: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:42:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:42:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-10T21:42:08.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:42:08.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:42:08.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c1236fc3-850e-470b-a918-785b96496ec5 2024-09-10T21:42:08.612 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 c1236fc3-850e-470b-a918-785b96496ec5 2024-09-10T21:42:08.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c1236fc3-850e-470b-a918-785b96496ec5' 2024-09-10T21:42:08.612 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:42:08.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwveBmU7hqJRAATb2JlEayxEFtj5mGimGWnQ== 2024-09-10T21:42:08.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwveBmU7hqJRAATb2JlEayxEFtj5mGimGWnQ=="}' 2024-09-10T21:42:08.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c1236fc3-850e-470b-a918-785b96496ec5 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:42:08.832 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:42:08.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:42:08.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAwveBmU7hqJRAATb2JlEayxEFtj5mGimGWnQ== --osd-uuid c1236fc3-850e-470b-a918-785b96496ec5 2024-09-10T21:42:08.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:08.874+0000 7faef9824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:08.883 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:08.882+0000 7faef9824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:08.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:08.883+0000 7faef9824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:08.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:08.883+0000 7faef9824640 -1 bdev(0x5561a4b51c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:42:08.884 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:08.884+0000 7faef9824640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:42:14.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:42:14.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:42:14.556 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:42:14.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:42:14.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:42:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:42:14.918 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:42:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-10T21:42:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:42:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:42:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:42:14.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:14.947+0000 7f34b90eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:14.950 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:14.949+0000 7f34b90eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:14.951 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:14.951+0000 7f34b90eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:15.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:42:15.194 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:42:15.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:15.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:15.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:16.318 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:16.317+0000 7f34b90eb640 -1 Falling back to public interface 2024-09-10T21:42:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:16.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:16.472 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:42:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:42:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:16.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:16.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:17.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:17.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:17.758 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:42:17.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:42:17.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:17.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:18.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:18.480 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:42:18.479+0000 7f34b90eb640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:42:19.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:19.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:19.046 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:42:19.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:42:19.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:19.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:19.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:20.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:20.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:20.398 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:42:20.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:42:20.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:20.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:20.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:21.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:21.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:21.686 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:42:21.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:42:21.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:21.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:21.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:22.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:22.974 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:42:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:42:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:22.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:23.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:24.265 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:42:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:42:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:24.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:25.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:25.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:25.554 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:42:25.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:42:25.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:25.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:25.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:26.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:26.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:26.844 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:42:26.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:42:26.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:26.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:27.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:28.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:42:28.218 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:42:28.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:28.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:28.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:29.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:29.507 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:42:29.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:42:29.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:29.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:29.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:30.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:30.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:30.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:42:30.794 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:42:30.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:30.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:31.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:32.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:32.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:32.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:42:32.304 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:42:32.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:32.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:33.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:33.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:33.808 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:42:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:42:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:34.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:35.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:35.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:35.502 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:42:35.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:42:35.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:35.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:36.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:42:37.119 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:42:37.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:37.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:37.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:38.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:38.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:38.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:42:38.536 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:42:38.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:38.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:38.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:39.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:39.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:39.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:42:39.919 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:42:39.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:39.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:40.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:42:41.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:42:41.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:42:41.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:42:41.207 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:42:41.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:42:41.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:42:41.484 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2868133094,v1:127.0.0.1:6803/2868133094] [v2:127.0.0.1:6804/2868133094,v1:127.0.0.1:6805/2868133094] exists,up c1236fc3-850e-470b-a918-785b96496ec5 2024-09-10T21:42:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:42:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:42:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:42:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-09-10T21:42:41.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2024-09-10T21:42:41.858 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool' created 2024-09-10T21:42:41.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:42:42.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: wait_for_clean 2024-09-10T21:42:42.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:42:42.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:42:42.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:42:42.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:42:42.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:42:42.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:42:42.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:42:42.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:42:42.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:42:42.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:42:42.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:42:42.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:42:42.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:42:42.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:42:42.979 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:42:43.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:42:43.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:42:43.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:42:43.252 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:42:43.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836486 2024-09-10T21:42:43.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836486 2024-09-10T21:42:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836486' 2024-09-10T21:42:43.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:42:43.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836486 2024-09-10T21:42:43.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:42:43.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:42:43.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836486 2024-09-10T21:42:43.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:42:43.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836486 2024-09-10T21:42:43.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-10T21:42:43.736 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836486 2024-09-10T21:42:43.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:42:44.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836486 2024-09-10T21:42:44.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:42:45.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:42:45.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:42:45.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-10T21:42:45.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:42:45.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:42:45.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:42:45.636 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:42:45.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-10T21:42:45.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:42:45.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:42:45.910 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:42:46.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local osd=0 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:42:46.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:42:46.617 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:42:46.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:42:47.145 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:42:47.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:42:47.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:42:47.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-09-10T21:42:47.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-09-10T21:42:47.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2024-09-10T21:42:47.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2024-09-10T21:42:47.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2024-09-10T21:42:47.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:42:47.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: local primary=0 2024-09-10T21:42:47.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-09-10T21:42:47.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2024-09-10T21:42:47.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2024-09-10T21:42:47.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2024-09-10T21:42:47.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:42:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local pg=1.1 2024-09-10T21:42:47.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: ceph osd dump 2024-09-10T21:42:47.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-09-10T21:42:47.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: awk '{ print $2 }' 2024-09-10T21:42:48.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: poolid=1 2024-09-10T21:42:48.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: local otherpgs 2024-09-10T21:42:48.228 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: expr 8 - 1 2024-09-10T21:42:48.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: seq 0 7 2024-09-10T21:42:48.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:48.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.0 2024-09-10T21:42:48.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-09-10T21:42:48.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-09-10T21:42:48.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:48.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:48.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:48.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:48.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:48.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:48.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-09-10T21:42:48.601 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:48.601 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:48.601 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:48.601 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:08.600748+0000" 2024-09-10T21:42:48.601 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:48.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:48.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.1 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: continue 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.2 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:48.611 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:48.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:48.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-09-10T21:42:48.998 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:48.999 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:48.999 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:48.999 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:08.998578+0000" 2024-09-10T21:42:48.999 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:49.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.3 2024-09-10T21:42:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-09-10T21:42:49.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-09-10T21:42:49.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:49.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:49.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:49.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:49.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:49.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:49.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-09-10T21:42:49.374 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:49.374 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:49.374 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:49.374 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:09.373969+0000" 2024-09-10T21:42:49.374 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.4 2024-09-10T21:42:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-09-10T21:42:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-09-10T21:42:49.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:49.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:49.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:49.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:49.386 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:49.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-09-10T21:42:49.770 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:49.771 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:49.771 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:49.771 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:09.770655+0000" 2024-09-10T21:42:49.771 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:49.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:49.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.5 2024-09-10T21:42:49.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-09-10T21:42:49.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-09-10T21:42:49.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:49.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:49.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:49.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:49.781 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:50.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:50.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-09-10T21:42:50.149 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:50.149 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:50.149 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:50.149 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:10.148877+0000" 2024-09-10T21:42:50.149 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:50.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:50.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.6 2024-09-10T21:42:50.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-09-10T21:42:50.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-09-10T21:42:50.158 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:50.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:50.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:50.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:50.159 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:50.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:50.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-09-10T21:42:50.526 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:50.526 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:50.526 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:50.526 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:10.526032+0000" 2024-09-10T21:42:50.527 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:50.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-10T21:42:50.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.7 2024-09-10T21:42:50.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-09-10T21:42:50.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-09-10T21:42:50.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:42:50.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:42:50.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:42:50.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:42:50.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:42:50.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:42:50.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-09-10T21:42:50.905 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:42:50.905 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:42:50.905 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:42:50.905 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:41:10.903820+0000" 2024-09-10T21:42:50.905 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:42:50.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: sleep 15 2024-09-10T21:43:05.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: flush_pg_stats 2024-09-10T21:43:05.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:43:05.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:43:06.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:43:06.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:43:06.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:43:06.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:43:06.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:43:06.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-10T21:43:06.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:43:06.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-10T21:43:06.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-10T21:43:06.306 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836492 2024-09-10T21:43:06.306 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:43:06.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-10T21:43:06.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:43:07.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:43:07.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:43:07.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-10T21:43:07.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-10T21:43:07.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:07.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:07.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:07.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:08.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:08.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-09-10T21:43:08.242 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:43:08.242 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:43:08.242 INFO:tasks.workunit.client.0.smithi181.stdout: "must": true, 2024-09-10T21:43:08.243 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "0.000000" 2024-09-10T21:43:08.243 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:43:08.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: ceph osd unset noscrub 2024-09-10T21:43:08.622 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is unset 2024-09-10T21:43:08.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-09-10T21:43:09.225 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is unset 2024-09-10T21:43:09.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6244: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:09.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-10T21:43:09.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:09.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:09.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:09.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:09.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-10T21:43:09.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:09.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:09.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:09.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:42:41.802447+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:09.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:10.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:10.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:10.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-10T21:43:10.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:10.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:10.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:10.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:10.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:42:41.802447+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:10.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:11.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:11.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:11.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-10T21:43:11.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:11.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:11.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:11.796 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:12.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:42:41.802447+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:12.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:13.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:13.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:13.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-10T21:43:13.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:13.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:13.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:13.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:09.045962+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:13.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:13.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:13.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:14.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:14.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:14.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:14.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:14.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:14.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:14.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:15.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:15.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:15.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:15.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:15.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:15.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:16.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:16.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:17.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:17.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:17.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:17.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:17.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:17.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:17.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:17.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:18.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:18.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:18.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:18.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:18.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:18.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:18.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:18.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:18.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:19.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:19.771 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:19.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:19.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:19.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:19.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:20.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:20.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:21.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:21.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:21.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:21.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:21.055 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:21.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:21.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:22.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:22.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:22.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:22.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:22.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:22.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:22.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:22.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:22.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:23.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:23.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:23.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:23.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:23.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:23.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:23.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:23.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:23.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:24.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:24.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:24.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:24.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:24.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:24.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:24.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:25.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:25.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:26.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:26.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:26.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:26.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:26.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:26.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:26.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:26.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:26.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:27.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:27.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:27.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:27.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:27.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:27.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:27.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:27.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:27.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:28.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:28.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:28.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:28.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:28.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:28.748 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:28.749 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:29.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:29.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:30.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:30.034 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:30.034 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:30.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:30.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:30.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:30.035 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:30.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:30.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:31.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:31.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:31.316 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:31.317 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:31.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:31.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:32.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:32.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:32.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:32.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:32.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:32.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:32.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:32.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:32.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:33.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:33.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:33.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:33.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:33.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:33.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:33.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:34.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:34.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:35.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:35.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:35.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:35.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:35.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:35.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:35.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:35.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:35.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:36.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:36.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:36.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:36.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:36.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:36.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:36.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:36.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:37.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:37.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:37.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:37.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:37.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:37.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:37.737 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:38.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:38.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:39.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:39.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:39.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:39.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:39.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:39.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:39.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:39.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:39.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:40.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:40.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:40.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:40.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:40.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:40.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:40.309 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:40.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:40.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:41.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:41.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:41.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:41.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:41.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:41.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:41.594 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:41.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:41.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:42.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:42.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:42.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:42.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:42.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:43.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:43.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:44.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:44.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:44.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:44.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:44.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:44.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:44.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:44.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:44.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:45.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:45.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:45.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:45.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:45.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:45.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:45.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:45.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:45.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:46.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:46.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:46.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:46.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:46.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:46.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:46.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:47.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:47.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:48.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:48.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:48.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:48.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:48.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:48.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:48.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:48.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:49.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:49.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:49.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:49.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:49.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:49.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:49.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:49.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:49.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:50.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:50.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:50.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:50.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:50.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:50.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:50.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:50.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:50.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:51.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:51.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:51.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:51.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:51.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:51.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:51.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:52.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:08.600748+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:52.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:53.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:43:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:43:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:53.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:43:53.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:49.866344+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.2 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:53.446 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-10T21:43:53.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2024-09-10T21:43:53.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:53.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:53.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:50.836712+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:53.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:43:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:43:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:53.719 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:43:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:09.373969+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:53.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:54.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:54.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:54.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:43:54.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:43:54.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:54.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:54.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:43:55.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:09.373969+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:55.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:56.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:43:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:43:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:56.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:43:56.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-03T21:41:09.373969+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:56.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:43:57.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:43:57.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:57.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-10T21:43:57.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:43:57.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:57.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:57.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:51.854803+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.4 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:57.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-10T21:43:57.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2024-09-10T21:43:57.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:57.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:57.838 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:52.887340+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.5 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-10T21:43:58.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-10T21:43:58.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:58.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:58.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:53.853852+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.6 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:58.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:54.819809+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.7 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:58.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:58.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-10T21:43:58.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:55.834541+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:43:58.927 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:43:09.045962+0000 '>' 2024-09-10T21:42:41.802447+0000 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -v purged_snaps 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: head -1 2024-09-10T21:43:59.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-09-10T21:43:59.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -q 1.1 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:43:59.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:43:59.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:43:59.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:43:59.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:43:59.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:43:59.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:43:59.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:43:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:43:59.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:43:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:43:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:43:59.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:43:59.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:43:59.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:43:59.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:43:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local poolname=psr_pool 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local objname=POBJ 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local scrubs=5 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local deep_scrubs=5 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local i1_day=86400 2024-09-10T21:43:59.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 86400 '*' 7 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 7}' 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local i7_days=604800 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 86400 '*' 14 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 14}' 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local i14_days=1209600 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5853: TEST_scrub_warning: local overdue=0.5 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-09-10T21:43:59.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5857: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:43:59.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:43:59.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:43:59.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-10T21:43:59.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:44:00.000 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:44:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:44:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:44:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:44:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:44:00.001 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:44:00.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:44:00.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:44:00.079 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:44:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:44:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:44:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:44:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:44:00.080 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:44:00.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:44:00.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-10T21:44:00.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:44:00.158 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:44:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:44:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:44:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:44:00.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:44:00.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:44:00.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:44:00.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:44:01.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:44:01.013 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:44:01.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11a5fcf1-da77-4e80-a198-c749aa098f25 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 11a5fcf1-da77-4e80-a198-c749aa098f25' 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 11a5fcf1-da77-4e80-a198-c749aa098f25 2024-09-10T21:44:01.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:44:01.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChveBmnwnHARAAEmjPU42Qxj6nPLE/wd6HFA== 2024-09-10T21:44:01.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChveBmnwnHARAAEmjPU42Qxj6nPLE/wd6HFA=="}' 2024-09-10T21:44:01.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11a5fcf1-da77-4e80-a198-c749aa098f25 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:44:01.239 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:44:01.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:44:01.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQChveBmnwnHARAAEmjPU42Qxj6nPLE/wd6HFA== --osd-uuid 11a5fcf1-da77-4e80-a198-c749aa098f25 2024-09-10T21:44:01.281 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:01.280+0000 7fecf6c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:01.282 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:01.282+0000 7fecf6c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:01.283 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:01.283+0000 7fecf6c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:01.283+0000 7fecf6c10640 -1 bdev(0x561b09081c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:44:01.284 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:01.283+0000 7fecf6c10640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:44:07.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:44:07.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:44:07.239 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:44:07.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:44:07.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:44:07.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:44:07.598 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:44:07.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-09-10T21:44:07.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:44:07.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:44:07.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:44:07.626 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:07.625+0000 7fc81280b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:07.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:07.627+0000 7fc81280b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:07.629 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:07.628+0000 7fc81280b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:44:07.872 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:44:07.873 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:44:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:07.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:08.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:09.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:09.003+0000 7fc81280b640 -1 Falling back to public interface 2024-09-10T21:44:09.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:09.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:09.150 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:44:09.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:44:09.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:09.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:09.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:10.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:10.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:10.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:44:10.440 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:44:10.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:10.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:10.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:44:10.579+0000 7fc81280b640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:44:10.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:11.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:11.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:11.757 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:44:11.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:44:11.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:11.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:12.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:13.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:13.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:13.044 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:44:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:44:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:13.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:13.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:14.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:14.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:14.367 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:44:14.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:44:14.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:14.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:14.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:15.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:15.655 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:44:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:44:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:15.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:15.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:16.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:16.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:16.941 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:44:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:44:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:16.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:17.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:18.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:18.320 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:44:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:44:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:18.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:19.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:19.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:19.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:44:19.608 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:44:19.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:19.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:19.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:20.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:20.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:20.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:44:20.896 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:44:20.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:20.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:21.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:22.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:22.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:22.186 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:44:22.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:44:22.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:22.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:22.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:23.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:23.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:23.745 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:44:23.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:44:23.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:23.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:24.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:25.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:44:25.293 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:44:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:25.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:25.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:26.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:26.929 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:44:26.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:44:26.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:26.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:27.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:28.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:28.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:28.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:44:28.626 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:44:28.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:28.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:29.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:30.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:44:30.294 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:44:30.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:30.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:30.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:31.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:31.916 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:44:31.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:31.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:44:31.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:31.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:32.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:44:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:44:33.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:44:33.301 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:44:33.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:44:33.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:44:33.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:44:33.583 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2718946501,v1:127.0.0.1:6803/2718946501] [v2:127.0.0.1:6804/2718946501,v1:127.0.0.1:6805/2718946501] exists,up 11a5fcf1-da77-4e80-a198-c749aa098f25 2024-09-10T21:44:33.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:44:33.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:44:33.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:44:33.584 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: expr 5 + 5 2024-09-10T21:44:33.585 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: seq 1 10 2024-09-10T21:44:33.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:33.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-09-10T21:44:33.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2024-09-10T21:44:34.006 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-1' created 2024-09-10T21:44:34.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:35.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:35.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:35.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:35.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:35.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:35.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:35.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:35.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:35.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:35.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:35.123 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:35.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:35.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:35.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:35.398 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:35.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-10T21:44:35.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-10T21:44:35.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-10T21:44:35.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:35.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-10T21:44:35.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:35.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:35.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-10T21:44:35.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:35.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-10T21:44:35.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-10T21:44:35.505 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836492 2024-09-10T21:44:35.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:35.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-10T21:44:35.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:36.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:44:36.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:37.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-10T21:44:37.069 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:37.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:37.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:44:37.431 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:44:37.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:44:37.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:44:37.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:44:37.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:37.705 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:38.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:44:38.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:44:38.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:44:38.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-10T21:44:38.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5867: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-09-10T21:44:38.439 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 scrub_max_interval to 1209600 2024-09-10T21:44:38.451 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:44:38.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 1 = 6 ']' 2024-09-10T21:44:38.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:38.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-09-10T21:44:38.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2024-09-10T21:44:38.967 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-2' created 2024-09-10T21:44:38.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:39.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:39.981 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:40.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:40.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:40.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:40.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:40.374 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:40.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-10T21:44:40.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-10T21:44:40.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-10T21:44:40.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:40.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-10T21:44:40.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:40.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:40.480 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-10T21:44:40.480 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:40.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-10T21:44:40.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-10T21:44:40.482 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836494 2024-09-10T21:44:40.482 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:40.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-10T21:44:40.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:41.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:44:41.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:42.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-10T21:44:42.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:42.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:42.059 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 2 == 0 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:44:42.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:44:42.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:44:42.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T21:44:42.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:44:42.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:42.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:43.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 2 2024-09-10T21:44:43.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:44:43.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:44:43.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-10T21:44:43.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:44:43.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 2 = 6 ']' 2024-09-10T21:44:43.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:43.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-09-10T21:44:43.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2024-09-10T21:44:43.408 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-3' created 2024-09-10T21:44:43.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:44.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:44.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:44.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:44.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:44.422 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:44.422 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:44.422 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:44.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:44.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:44.423 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:44.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:44.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:44.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:44.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:44.797 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:44.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-10T21:44:44.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-10T21:44:44.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-10T21:44:44.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:44.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-10T21:44:44.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:44.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:44.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-10T21:44:44.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:44.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-10T21:44:44.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-10T21:44:44.903 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836496 2024-09-10T21:44:44.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:45.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-10T21:44:45.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:45.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:45.204 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:44:45.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:44:45.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:44:45.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T21:44:45.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:44:45.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:45.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:46.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-10T21:44:46.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:44:46.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:44:46.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-10T21:44:46.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:44:46.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 3 = 6 ']' 2024-09-10T21:44:46.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:46.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-09-10T21:44:46.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2024-09-10T21:44:46.600 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-4' created 2024-09-10T21:44:46.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:47.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:47.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:47.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:47.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:47.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:47.614 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:47.615 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:47.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:47.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:47.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:47.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:47.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:47.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:47.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:47.987 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:48.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-10T21:44:48.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-10T21:44:48.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-10T21:44:48.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:48.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-10T21:44:48.090 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:48.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:48.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-10T21:44:48.092 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:48.093 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836498 2024-09-10T21:44:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-10T21:44:48.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-10T21:44:48.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:48.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-10T21:44:48.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:49.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:44:49.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:49.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-10T21:44:49.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:49.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:49.661 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:50.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:44:50.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:44:50.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:44:50.006 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:44:50.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:44:50.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:44:50.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:44:50.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:44:50.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:44:50.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:44:50.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:50.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:50.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:44:50.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:44:50.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:44:50.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-10T21:44:50.644 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:44:50.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 4 = 6 ']' 2024-09-10T21:44:50.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:50.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-09-10T21:44:50.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2024-09-10T21:44:51.017 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-5' created 2024-09-10T21:44:51.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:52.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:52.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:52.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:52.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:52.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:52.031 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:52.031 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:52.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:52.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:52.032 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:52.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:52.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:52.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:52.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:52.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:52.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-10T21:44:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-10T21:44:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-10T21:44:52.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:52.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-10T21:44:52.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:52.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:52.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-10T21:44:52.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-10T21:44:52.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-10T21:44:52.517 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836499 2024-09-10T21:44:52.517 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:52.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836499 2024-09-10T21:44:52.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:53.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:44:53.791 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:54.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-10T21:44:54.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:54.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:54.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:54.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T21:44:54.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:44:54.420 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:44:54.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T21:44:54.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:44:54.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:54.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:44:55.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T21:44:55.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:44:55.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:44:55.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-10T21:44:55.040 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:44:55.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 5 = 6 ']' 2024-09-10T21:44:55.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:44:55.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-09-10T21:44:55.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2024-09-10T21:44:55.493 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-6' created 2024-09-10T21:44:55.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:44:56.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:44:56.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:44:56.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:44:56.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:44:56.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:44:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:44:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:44:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:44:56.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:44:56.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:44:56.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:44:56.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:44:56.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:44:56.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:44:56.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:44:56.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T21:44:56.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T21:44:56.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T21:44:56.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:44:56.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T21:44:56.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:44:56.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:44:56.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T21:44:56.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:44:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T21:44:56.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T21:44:56.999 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836501 2024-09-10T21:44:56.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:57.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T21:44:57.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:58.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:44:58.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:58.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-10T21:44:58.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:44:59.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:44:59.561 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:44:59.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-10T21:44:59.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:44:59.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:44:59.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:00.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-10T21:45:00.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:45:00.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:45:00.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-10T21:45:00.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:45:00.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:00.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-10T21:45:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:45:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:45:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 6 = 1 ']' 2024-09-10T21:45:00.810 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:00.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-10T21:45:00.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5871: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-09-10T21:45:01.236 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 6 deep_scrub_interval to 1209600 2024-09-10T21:45:01.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:01.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-09-10T21:45:01.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2024-09-10T21:45:01.755 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-7' created 2024-09-10T21:45:01.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:45:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:45:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:45:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:45:02.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:45:02.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:45:02.876 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:45:03.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:45:03.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:45:03.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:45:03.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-10T21:45:03.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:45:03.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:45:03.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-10T21:45:03.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:45:03.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-10T21:45:03.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-10T21:45:03.288 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836504 2024-09-10T21:45:03.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-10T21:45:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:45:04.563 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-10T21:45:04.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:05.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:45:05.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:06.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-10T21:45:06.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:45:06.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:06.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:06.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 7 == 0 2024-09-10T21:45:06.482 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:45:06.482 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:45:06.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:45:06.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:45:06.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:45:06.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:45:06.483 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:45:06.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=7 2024-09-10T21:45:06.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:45:06.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:06.754 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:07.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 = 7 2024-09-10T21:45:07.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:45:07.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:45:07.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 7 = 1 ']' 2024-09-10T21:45:07.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:07.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-10T21:45:07.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:07.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-09-10T21:45:07.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2024-09-10T21:45:07.498 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-8' created 2024-09-10T21:45:07.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:45:08.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:45:08.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:45:08.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:45:08.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:45:08.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:45:08.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:45:08.514 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:45:08.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:45:08.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:45:08.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:45:08.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:45:08.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:45:08.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:45:08.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:45:08.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:45:08.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:45:08.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:45:08.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:45:08.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:45:08.896 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:45:08.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T21:45:08.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T21:45:08.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T21:45:08.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:45:08.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T21:45:08.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:45:09.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:45:09.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T21:45:09.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:45:09.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T21:45:09.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T21:45:09.002 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836506 2024-09-10T21:45:09.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:09.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-10T21:45:09.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:10.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:45:10.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:10.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-10T21:45:10.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:11.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:45:11.568 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:11.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T21:45:11.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:45:11.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:11.857 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-10T21:45:12.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:45:12.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:45:12.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:45:12.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:45:12.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:45:12.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:45:12.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:45:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-10T21:45:12.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:45:12.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:12.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-10T21:45:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:45:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:45:12.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 8 = 1 ']' 2024-09-10T21:45:12.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:12.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-10T21:45:12.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:12.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-09-10T21:45:12.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2024-09-10T21:45:13.234 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-9' created 2024-09-10T21:45:13.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:45:14.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:45:14.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:45:14.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:45:14.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:45:14.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:45:14.354 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:45:14.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:45:14.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:45:14.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:45:14.634 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:45:14.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-10T21:45:14.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-10T21:45:14.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-10T21:45:14.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:45:14.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-10T21:45:14.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:45:14.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:45:14.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-10T21:45:14.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:45:14.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-10T21:45:14.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-10T21:45:14.741 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836508 2024-09-10T21:45:14.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:15.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-10T21:45:15.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:16.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:45:16.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:16.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-10T21:45:16.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:45:16.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:16.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 9 == 0 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:45:16.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:45:16.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=9 2024-09-10T21:45:16.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:45:16.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:16.941 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:17.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 = 9 2024-09-10T21:45:17.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:45:17.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:45:17.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 9 = 1 ']' 2024-09-10T21:45:17.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:17.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-10T21:45:17.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:17.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-09-10T21:45:17.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2024-09-10T21:45:17.709 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'psr_pool-10' created 2024-09-10T21:45:17.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:45:18.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-10T21:45:18.722 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:45:18.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:45:18.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:45:18.723 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:45:18.724 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:45:18.724 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:45:18.724 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:45:18.724 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:45:18.724 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:45:18.830 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:45:19.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:45:19.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:45:19.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:45:19.118 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:45:19.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-10T21:45:19.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-10T21:45:19.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-10T21:45:19.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:45:19.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-10T21:45:19.222 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:45:19.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:45:19.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-10T21:45:19.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:45:19.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-10T21:45:19.224 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836510 2024-09-10T21:45:19.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-10T21:45:19.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:19.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-10T21:45:19.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:20.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:45:20.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:20.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-10T21:45:20.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:21.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:45:21.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:22.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-10T21:45:22.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:45:22.093 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:22.094 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:22.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 10 == 0 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:45:22.447 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:45:22.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=10 2024-09-10T21:45:22.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:45:22.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:45:22.707 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:45:23.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 = 10 2024-09-10T21:45:23.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:45:23.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:45:23.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 10 = 1 ']' 2024-09-10T21:45:23.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:23.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-10T21:45:23.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: local primary=0 2024-09-10T21:45:23.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph osd set noscrub 2024-09-10T21:45:23.469 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:45:23.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-09-10T21:45:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:45:23.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-09-10T21:45:24.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-09-10T21:45:24.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-09-10T21:45:24.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-09-10T21:45:25.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: seq 1 5 2024-09-10T21:45:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-10T21:45:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-10T21:45:25.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: TEST_scrub_warning: overdue_seconds=1900800 2024-09-10T21:45:25.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 1900800 + 100 2024-09-10T21:45:25.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-09-10T21:45:25.331 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.332 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:45:25.332 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.332 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-19T21:42:05.331242+0000" 2024-09-10T21:45:25.332 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-10T21:45:25.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-10T21:45:25.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 200 2024-09-10T21:45:25.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-09-10T21:45:25.438 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.438 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:45:25.438 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.438 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:40:25.438107+0000" 2024-09-10T21:45:25.438 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-10T21:45:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-10T21:45:25.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.448 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 300 2024-09-10T21:45:25.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-09-10T21:45:25.546 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.546 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:45:25.546 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.547 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:38:45.546326+0000" 2024-09-10T21:45:25.547 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-10T21:45:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-10T21:45:25.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 400 2024-09-10T21:45:25.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-09-10T21:45:25.655 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.655 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:45:25.655 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.655 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:37:05.654697+0000" 2024-09-10T21:45:25.655 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-10T21:45:25.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-10T21:45:25.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 500 2024-09-10T21:45:25.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-09-10T21:45:25.763 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.763 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:45:25.763 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.763 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:35:25.762787+0000" 2024-09-10T21:45:25.763 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.772 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:25.773 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 5 2024-09-10T21:45:25.774 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: seq 6 10 2024-09-10T21:45:25.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:25.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:25.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-10T21:45:25.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5901: TEST_scrub_warning: overdue_seconds=1900800 2024-09-10T21:45:25.777 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 1900800 + 600 2024-09-10T21:45:25.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-09-10T21:45:25.875 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.875 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T21:45:25.875 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.875 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-19T21:33:45.875273+0000" 2024-09-10T21:45:25.876 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:25.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-10T21:45:25.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.886 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 700 2024-09-10T21:45:25.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-09-10T21:45:25.985 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:25.985 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T21:45:25.985 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:25.985 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:32:05.984514+0000" 2024-09-10T21:45:25.985 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:25.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:25.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:25.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-10T21:45:25.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:25.995 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 800 2024-09-10T21:45:25.996 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-09-10T21:45:26.093 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:26.093 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T21:45:26.093 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:26.093 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:30:26.092767+0000" 2024-09-10T21:45:26.093 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:26.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:26.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:26.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-10T21:45:26.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:26.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 900 2024-09-10T21:45:26.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-09-10T21:45:26.201 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:26.201 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T21:45:26.201 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:26.201 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:28:46.201061+0000" 2024-09-10T21:45:26.201 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:26.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-10T21:45:26.210 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-10T21:45:26.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-10T21:45:26.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-10T21:45:26.212 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 1000 2024-09-10T21:45:26.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-09-10T21:45:26.310 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:45:26.310 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T21:45:26.310 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:45:26.310 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-30T09:27:06.309844+0000" 2024-09-10T21:45:26.310 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:45:26.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: flush_pg_stats 2024-09-10T21:45:26.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:45:26.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:45:26.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:45:26.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:45:26.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:45:26.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:45:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-10T21:45:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-10T21:45:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-10T21:45:26.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:45:26.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-10T21:45:26.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:45:26.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:45:26.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-10T21:45:26.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:45:26.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-10T21:45:26.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-10T21:45:26.697 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836512 2024-09-10T21:45:26.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:26.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-10T21:45:26.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:45:27.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:45:27.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:45:28.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-10T21:45:28.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-09-10T21:45:28.566 INFO:tasks.workunit.client.0.smithi181.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-10T21:45:28.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: ceph health detail 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout: pg 10.0 not deep-scrubbed since 2024-08-30T09:27:06.309844+0000 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout: pg 9.0 not deep-scrubbed since 2024-08-30T09:28:46.201061+0000 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout: pg 8.0 not deep-scrubbed since 2024-08-30T09:30:26.092767+0000 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout: pg 7.0 not deep-scrubbed since 2024-08-30T09:32:05.984514+0000 2024-09-10T21:45:28.905 INFO:tasks.workunit.client.0.smithi181.stdout: pg 6.0 not deep-scrubbed since 2024-08-19T21:33:45.875273+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 10.0 not scrubbed since 2024-08-30T09:27:06.309844+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 9.0 not scrubbed since 2024-08-30T09:28:46.201061+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 8.0 not scrubbed since 2024-08-30T09:30:26.092767+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 7.0 not scrubbed since 2024-08-30T09:32:05.984514+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 6.0 not scrubbed since 2024-08-19T21:33:45.875273+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 5.0 not scrubbed since 2024-08-30T09:35:25.762787+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 4.0 not scrubbed since 2024-08-30T09:37:05.654697+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 3.0 not scrubbed since 2024-08-30T09:38:45.546326+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 2.0 not scrubbed since 2024-08-30T09:40:25.438107+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout: pg 1.0 not scrubbed since 2024-08-19T21:42:05.331242+0000 2024-09-10T21:45:28.906 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-1' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-2' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-3' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-4' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-5' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-6' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-7' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-8' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-9' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: application not enabled on pool 'psr_pool-10' 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-1' has no replicas configured 2024-09-10T21:45:28.907 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-2' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-3' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-4' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-5' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-6' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-7' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-8' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-9' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout: pool 'psr_pool-10' has no replicas configured 2024-09-10T21:45:28.908 INFO:tasks.workunit.client.0.smithi181.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-09-10T21:45:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: ceph health 2024-09-10T21:45:28.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-09-10T21:45:29.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph health 2024-09-10T21:45:29.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-09-10T21:45:29.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: ceph health detail 2024-09-10T21:45:29.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: grep 'not scrubbed since' 2024-09-10T21:45:29.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: wc -l 2024-09-10T21:45:29.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: COUNT=10 2024-09-10T21:45:29.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 5+5 2024-09-10T21:45:29.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-09-10T21:45:29.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph health detail 2024-09-10T21:45:29.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-09-10T21:45:29.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: wc -l 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: COUNT=5 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5924: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:45:30.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:45:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:45:30.345 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:45:30.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:45:30.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:45:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:45:30.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:45:30.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:45:30.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:45:30.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:45:30.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:45:30.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:45:30.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:45:30.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:45:30.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:45:30.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:45:30.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:30.361 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:30.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:45:30.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:45:30.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:45:30.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:45:30.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:45:30.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:45:30.365 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:45:30.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:45:30.366 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:45:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:45:30.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:45:30.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:45:30.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:45:30.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:45:30.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:45:30.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:45:30.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:45:30.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:45:30.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:45:30.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:45:30.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:30.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:30.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:45:30.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:45:30.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:45:30.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:45:30.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:45:30.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:30.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:30.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-10T21:45:30.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:45:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:45:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:45:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:45:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:45:30.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-10T21:45:30.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:45:30.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:45:30.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:45:30.831 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:45:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:30.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:45:30.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:45:31.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:45:31.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:45:31.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:45:31.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:45:31.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:45:31.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:45:31.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:31.106 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:45:31.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:45:31.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:45:31.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:31.992 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:45:31.993 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:45:31.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:45:32.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:45:32.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:45:32.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:45:32.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:45:32.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:45:32.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:45:32.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:45:32.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a524aecf-0edf-49f1-9032-e9fcf2176abc 2024-09-10T21:45:32.028 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 a524aecf-0edf-49f1-9032-e9fcf2176abc 2024-09-10T21:45:32.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a524aecf-0edf-49f1-9032-e9fcf2176abc' 2024-09-10T21:45:32.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:45:32.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8veBmyTWbAhAA9Qq5gUebtiSd9nNYvLvGjA== 2024-09-10T21:45:32.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8veBmyTWbAhAA9Qq5gUebtiSd9nNYvLvGjA=="}' 2024-09-10T21:45:32.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a524aecf-0edf-49f1-9032-e9fcf2176abc -i td/osd-scrub-repair/0/new.json 2024-09-10T21:45:32.289 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:45:32.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:45:32.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQD8veBmyTWbAhAA9Qq5gUebtiSd9nNYvLvGjA== --osd-uuid a524aecf-0edf-49f1-9032-e9fcf2176abc 2024-09-10T21:45:32.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:32.331+0000 7fa27109c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:32.340 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:32.339+0000 7fa27109c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:32.341 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:32.340+0000 7fa27109c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:32.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:32.341+0000 7fa27109c640 -1 bdev(0x5618dab3bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:45:32.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:32.341+0000 7fa27109c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:45:38.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:45:38.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:45:38.622 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:45:38.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:45:38.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:45:38.984 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:45:38.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:45:38.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:45:38.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T21:45:38.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:45:38.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:45:39.014 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:39.013+0000 7fdfc0dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:39.016 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:39.015+0000 7fdfc0dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:39.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:39.016+0000 7fdfc0dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:45:39.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:45:39.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:45:39.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:45:39.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:39.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:39.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:40.369 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:40.368+0000 7fdfc0dc5640 -1 Falling back to public interface 2024-09-10T21:45:40.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:40.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:40.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:45:40.539 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:45:40.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:40.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:40.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:41.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:41.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:41.825 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:45:41.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:45:41.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:41.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:42.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:42.530 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:45:42.529+0000 7fdfc0dc5640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:45:43.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:43.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:43.112 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:45:43.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:45:43.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:43.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:43.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:44.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:44.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:44.449 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:45:44.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:45:44.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:44.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:44.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:45.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:45.736 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:45:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:45:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:45.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:46.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:47.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:47.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:47.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:45:47.024 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:45:47.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:47.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:47.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:48.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:48.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:48.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:45:48.320 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:45:48.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:48.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:48.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:49.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:49.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:49.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:45:49.609 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:45:49.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:49.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:49.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:50.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:50.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:50.899 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:45:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:45:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:50.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:51.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:52.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:52.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:52.259 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:45:52.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:45:52.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:52.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:52.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:45:53.551 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:45:53.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:53.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:53.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:54.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:54.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:54.838 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:45:54.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:45:54.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:54.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:55.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:56.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:56.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:56.502 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:45:56.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:45:56.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:56.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:57.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:58.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:58.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:58.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:45:58.074 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:45:58.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:58.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:58.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:45:59.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:45:59.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:45:59.540 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:45:59.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:45:59.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:45:59.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:45:59.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:00.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:00.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:00.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:46:00.949 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:46:00.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:00.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:01.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:02.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:02.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:02.646 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:46:02.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:46:02.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:02.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:03.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:04.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:46:04.047 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:46:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:04.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:04.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:05.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:05.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:05.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:46:05.594 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:46:05.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:05.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:05.943 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1635833172,v1:127.0.0.1:6803/1635833172] [v2:127.0.0.1:6804/1635833172,v1:127.0.0.1:6805/1635833172] exists,up a524aecf-0edf-49f1-9032-e9fcf2176abc 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:46:05.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:46:05.947 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:46:05.948 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:46:05.949 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:46:05.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=517ca436-1a41-4425-a773-5fd6ac5f9e59 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 517ca436-1a41-4425-a773-5fd6ac5f9e59' 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 517ca436-1a41-4425-a773-5fd6ac5f9e59 2024-09-10T21:46:05.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:46:05.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdvuBms+STORAAGYw4kHMXOzz9Cc5ZUSLCnQ== 2024-09-10T21:46:05.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdvuBms+STORAAGYw4kHMXOzz9Cc5ZUSLCnQ=="}' 2024-09-10T21:46:05.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 517ca436-1a41-4425-a773-5fd6ac5f9e59 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:46:06.340 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:46:06.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:46:06.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAdvuBms+STORAAGYw4kHMXOzz9Cc5ZUSLCnQ== --osd-uuid 517ca436-1a41-4425-a773-5fd6ac5f9e59 2024-09-10T21:46:06.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:06.383+0000 7f952451b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:06.425 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:06.423+0000 7f952451b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:06.424+0000 7f952451b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:06.425+0000 7f952451b640 -1 bdev(0x561124c71c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:46:06.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:06.425+0000 7f952451b640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:46:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:46:12.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:46:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:46:12.058 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:46:12.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:46:12.419 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:46:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:46:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:46:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-10T21:46:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:46:12.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:46:12.450 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:12.449+0000 7f108380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:12.451 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:12.451+0000 7f108380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:12.453 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:12.452+0000 7f108380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:46:12.694 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:46:12.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:12.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:12.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:13.809 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:13.808+0000 7f108380b640 -1 Falling back to public interface 2024-09-10T21:46:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:13.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:46:13.971 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:46:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:13.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:14.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:15.122 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:15.121+0000 7f108380b640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:46:15.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:15.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:15.263 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:46:15.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:46:15.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:15.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:15.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:16.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:16.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:16.654 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:46:16.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:46:16.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:16.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:16.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:17.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:17.976 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:46:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:46:17.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:17.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:18.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:19.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:19.325 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:46:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:46:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:19.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:19.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:20.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:20.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:20.611 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:46:20.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:46:20.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:20.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:20.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:21.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:21.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:21.972 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:46:21.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:46:21.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:21.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:22.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:23.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:23.262 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:46:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:46:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:23.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:23.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:24.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:24.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:24.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:46:24.553 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:46:24.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:24.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:24.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:25.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:25.839 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:46:25.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:25.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:46:25.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:25.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:26.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:27.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:27.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:27.184 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:46:27.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:46:27.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:27.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:27.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:28.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:28.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:28.469 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:46:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:46:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:28.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:29.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:30.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:30.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:30.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:46:30.476 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:46:30.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:30.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:31.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:32.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:32.261 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:46:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:46:32.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:32.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:33.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:34.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:34.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:34.004 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:46:34.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:46:34.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:34.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:35.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:36.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:46:36.169 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:46:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:36.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:36.926 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:37.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:37.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:37.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:46:37.928 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:46:37.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:37.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:38.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:39.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:39.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:39.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:46:39.247 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:46:39.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:39.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:46:39.542 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2099537689,v1:127.0.0.1:6811/2099537689] [v2:127.0.0.1:6812/2099537689,v1:127.0.0.1:6813/2099537689] exists,up 517ca436-1a41-4425-a773-5fd6ac5f9e59 2024-09-10T21:46:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:46:39.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:46:39.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:46:39.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-09-10T21:46:39.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T21:46:39.808 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' does not exist 2024-09-10T21:46:39.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T21:46:39.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T21:46:40.256 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'rbd' created 2024-09-10T21:46:40.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:46:41.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T21:46:43.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-09-10T21:46:43.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:46:43.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:46:43.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:46:43.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:46:43.111 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:46:43.111 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:46:43.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:46:43.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:46:43.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:46:43.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:46:43.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:46:43.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:46:43.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:46:43.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:46:43.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:46:43.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:46:43.490 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:46:43.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:46:43.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:46:43.491 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:46:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-10T21:46:43.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-10T21:46:43.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-10T21:46:43.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:46:43.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:46:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-10T21:46:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-10T21:46:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672966' 2024-09-10T21:46:43.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:46:43.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-10T21:46:43.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:46:43.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:46:43.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-10T21:46:43.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:46:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-10T21:46:43.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-10T21:46:43.715 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836492 2024-09-10T21:46:43.715 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:46:44.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-10T21:46:44.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:46:45.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:46:45.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:46:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-10T21:46:45.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:46:45.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-10T21:46:45.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:46:45.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:46:45.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-10T21:46:45.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:46:45.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-10T21:46:45.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-10T21:46:45.298 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672966 2024-09-10T21:46:45.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:46:45.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-10T21:46:45.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:46:45.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:46:45.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:46:45.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:46:45.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:46:45.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:46:45.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:46:45.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:46:45.915 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:46:45.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:46:45.916 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:46:46.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:46:46.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:46:46.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:46:46.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:46:46.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:46:46.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:46:46.900 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:46:46.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:46:47.445 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:46:47.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:46:47.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:46:47.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T21:46:47.884 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:46:48.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:46:48.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-10T21:46:48.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:46:48.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-09-10T21:46:48.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-09-10T21:46:48.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-10T21:46:48.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:46:48.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-09-10T21:46:48.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-10T21:46:48.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:46:48.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:46:48.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:46:48.746 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:46:49.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-09-10T21:46:40.140569+0000 2024-09-10T21:46:49.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-09-10T21:46:49.131 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:46:49.131 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T21:46:49.131 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T21:46:49.131 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T21:45:09.130882+0000" 2024-09-10T21:46:49.131 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:46:49.139 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-09-10T21:46:49.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-10T21:46:49.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-10T21:46:49.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:46:49.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:46:49.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:46:49.141 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:46:49.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-10T21:46:40.140569+0000 '>' 2024-09-10T21:46:40.140569+0000 2024-09-10T21:46:49.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-10T21:46:50.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-10T21:46:50.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-10T21:46:50.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-10T21:46:50.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:46:50.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:46:50.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:46:50.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:46:50.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-10T21:46:40.140569+0000 '>' 2024-09-10T21:46:40.140569+0000 2024-09-10T21:46:50.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-10T21:46:51.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-10T21:46:51.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-10T21:46:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-10T21:46:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-10T21:46:51.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:46:51.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:46:51.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-10T21:46:51.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-10T21:46:40.140569+0000 '>' 2024-09-10T21:46:40.140569+0000 2024-09-10T21:46:51.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-10T21:46:52.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-10T21:46:52.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-10T21:46:52.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T21:46:52.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:46:52.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:46:52.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:46:53.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-10T21:46:53.746 INFO:tasks.workunit.client.0.smithi181.stdout:_ 2024-09-10T21:46:53.747 INFO:tasks.workunit.client.0.smithi181.stdout:snapset 2024-09-10T21:46:54.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:46:54.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:46:54.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:46:54.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:46:54.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:46:54.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:46:54.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:start osd.0 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:46:54.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:46:54.048 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:54.046+0000 7fb3da20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:54.050 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:54.048+0000 7fb3da20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:54.051 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:54.049+0000 7fb3da20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:46:54.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:54.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:54.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:55.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:55.400+0000 7fb3da20f640 -1 Falling back to public interface 2024-09-10T21:46:55.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:55.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:55.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:46:55.648 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:46:55.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:55.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:55.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:56.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:56.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:56.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:46:56.935 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:46:56.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:56.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:57.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:46:57.359 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:46:57.358+0000 7fb3da20f640 -1 osd.0 22 log_to_monitors true 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:3 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:46:58.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:46:58.662 INFO:tasks.workunit.client.0.smithi181.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2475198096,v1:127.0.0.1:6803/2475198096] [v2:127.0.0.1:6804/2475198096,v1:127.0.0.1:6805/2475198096] exists,up a524aecf-0edf-49f1-9032-e9fcf2176abc 2024-09-10T21:46:58.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:46:58.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:46:58.663 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:46:58.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:46:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:46:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:46:59.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:46:59.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:46:59.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:46:59.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-10T21:46:59.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-10T21:46:59.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-10T21:46:59.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:46:59.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:46:59.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-10T21:46:59.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-10T21:46:59.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-09-10T21:46:59.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:46:59.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-10T21:46:59.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:46:59.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:46:59.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-10T21:46:59.341 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:46:59.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-10T21:46:59.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-10T21:46:59.342 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.0 seq 111669149698 2024-09-10T21:46:59.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:46:59.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-10T21:46:59.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:47:00.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:47:00.621 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:47:00.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-10T21:47:00.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:47:00.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-10T21:47:00.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:47:00.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:47:00.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-10T21:47:00.907 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:47:00.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-10T21:47:00.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-10T21:47:00.909 INFO:tasks.workunit.client.0.smithi181.stderr:waiting osd.1 seq 42949672970 2024-09-10T21:47:00.909 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:47:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-10T21:47:01.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:47:01.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:47:01.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:47:01.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T21:47:01.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:47:01.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:47:01.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T21:47:01.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:47:01.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:47:01.795 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:47:02.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T21:47:02.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:47:02.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:47:02.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-10T21:47:02.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-10T21:47:02.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:47:02.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:47:02.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:47:02.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:47:02.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:47:02.324 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:47:02.325 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:47:02.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:47:02.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:47:02.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:47:02.327 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:47:02.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:47:02.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:47:02.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:47:02.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:47:02.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:47:02.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:47:02.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:47:02.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:47:02.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:02.342 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:02.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:47:02.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:47:02.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:47:02.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:47:02.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:47:02.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:47:02.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:47:02.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:47:02.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:47:02.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:47:02.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:47:02.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:47:02.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:47:02.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:47:02.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:47:02.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:47:02.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:47:02.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:47:02.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:02.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:02.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:47:02.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:47:02.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:47:02.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:47:02.355 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:47:02.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:02.356 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:02.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:47:02.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:47:02.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:47:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:47:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:47:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-09-10T21:47:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-09-10T21:47:02.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:47:02.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:02.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:47:02.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:47:02.959 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:47:02.960 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:47:03.041 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:47:03.042 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:47:03.042 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:47:03.042 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:47:03.043 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:03.043 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:03.043 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:47:03.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:47:03.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:47:03.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:03.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:03.936 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:47:03.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:47:03.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-10T21:47:03.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:47:03.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-10T21:47:03.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:47:03.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:47:03.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:47:03.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:47:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:47:03.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:47:03.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:47:03.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:47:03.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:47:03.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:47:03.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bdf0285d-2f61-4ff2-a58f-0933e9a52792 2024-09-10T21:47:03.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bdf0285d-2f61-4ff2-a58f-0933e9a52792' 2024-09-10T21:47:03.978 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 bdf0285d-2f61-4ff2-a58f-0933e9a52792 2024-09-10T21:47:03.978 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:47:03.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXvuBmtVzfOhAAd9xPhAD+nLF9dACn1rdKJg== 2024-09-10T21:47:03.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXvuBmtVzfOhAAd9xPhAD+nLF9dACn1rdKJg=="}' 2024-09-10T21:47:03.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bdf0285d-2f61-4ff2-a58f-0933e9a52792 -i td/osd-scrub-repair/0/new.json 2024-09-10T21:47:04.190 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:47:04.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:47:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXvuBmtVzfOhAAd9xPhAD+nLF9dACn1rdKJg== --osd-uuid bdf0285d-2f61-4ff2-a58f-0933e9a52792 2024-09-10T21:47:04.230 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:04.230+0000 7fa241487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:04.241 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:04.240+0000 7fa241487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:04.241+0000 7fa241487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:04.242+0000 7fa241487640 -1 bdev(0x5563e6163c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:47:04.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:04.242+0000 7fa241487640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:47:09.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:47:09.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:47:09.999 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:47:09.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:47:09.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:47:10.362 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:47:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:47:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:47:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:47:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:47:10.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:47:10.390 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:10.389+0000 7fe9a4a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:10.392 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:10.391+0000 7fe9a4a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:10.393 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:10.392+0000 7fe9a4a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:10.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:47:10.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:47:10.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:47:10.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:47:10.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:47:10.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:10.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:47:10.639 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:47:10.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:10.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:10.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:11.513 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:11.512+0000 7fe9a4a0f640 -1 Falling back to public interface 2024-09-10T21:47:11.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:11.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:11.917 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:47:11.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:47:11.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:11.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:12.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:13.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:13.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:13.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:47:13.205 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:47:13.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:13.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:13.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:13.330+0000 7fe9a4a0f640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:47:13.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:14.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:14.507 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:47:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:47:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:14.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:14.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:15.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:15.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:15.797 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:47:15.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:47:15.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:15.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:16.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:17.086 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:47:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:47:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:17.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:18.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:18.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:18.376 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:47:18.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:47:18.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:18.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:18.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:19.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:19.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:19.664 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:47:19.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:47:19.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:19.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:19.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:20.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:20.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:20.947 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:47:20.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:47:20.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:20.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:21.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:22.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:22.246 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:47:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:47:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:22.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:22.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:23.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:23.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:23.533 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:47:23.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:47:23.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:23.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:23.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:24.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:24.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:24.846 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:47:24.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:47:24.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:24.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:25.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:26.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:26.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:26.456 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:47:26.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:47:26.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:26.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:27.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:28.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:28.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:28.035 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:47:28.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:47:28.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:28.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:28.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:29.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:29.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:29.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:47:29.806 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:47:29.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:29.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:31.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:31.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:31.236 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:47:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:47:31.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:31.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:31.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:32.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:32.844 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:47:32.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:47:32.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:32.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:33.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:34.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:34.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:34.504 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:47:34.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:47:34.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:34.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:35.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:36.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:36.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:36.248 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:47:36.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:47:36.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:36.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:47:36.532 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/873800364,v1:127.0.0.1:6803/873800364] [v2:127.0.0.1:6804/873800364,v1:127.0.0.1:6805/873800364] exists,up bdf0285d-2f61-4ff2-a58f-0933e9a52792 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:47:36.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:47:36.537 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:47:36.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff72e0b6-d830-4cee-9ef7-23bbfd1f394e 2024-09-10T21:47:36.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ff72e0b6-d830-4cee-9ef7-23bbfd1f394e' 2024-09-10T21:47:36.540 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 ff72e0b6-d830-4cee-9ef7-23bbfd1f394e 2024-09-10T21:47:36.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:47:36.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4vuBmDY0dIRAAkp1oUIoALWxfFdEiLbZtuQ== 2024-09-10T21:47:36.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4vuBmDY0dIRAAkp1oUIoALWxfFdEiLbZtuQ=="}' 2024-09-10T21:47:36.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff72e0b6-d830-4cee-9ef7-23bbfd1f394e -i td/osd-scrub-repair/1/new.json 2024-09-10T21:47:36.899 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:47:36.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:47:36.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4vuBmDY0dIRAAkp1oUIoALWxfFdEiLbZtuQ== --osd-uuid ff72e0b6-d830-4cee-9ef7-23bbfd1f394e 2024-09-10T21:47:36.940 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:36.939+0000 7f28f9e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:36.979 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:36.978+0000 7f28f9e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:36.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:36.979+0000 7f28f9e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:36.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:36.980+0000 7f28f9e13640 -1 bdev(0x559608f93c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:47:36.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:36.980+0000 7f28f9e13640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:47:42.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:47:42.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:47:42.999 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:47:42.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:47:42.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:47:43.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:47:43.358 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:47:43.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:47:43.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:47:43.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:47:43.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:47:43.385 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:43.384+0000 7fc88abb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:43.387 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:43.386+0000 7fc88abb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:43.388 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:43.387+0000 7fc88abb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:43.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:47:43.636 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:47:43.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:43.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:43.908 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:44.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:44.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:44.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:47:44.910 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:47:44.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:44.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:45.004 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:45.003+0000 7fc88abb9640 -1 Falling back to public interface 2024-09-10T21:47:45.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:46.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:46.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:46.198 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:47:46.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:47:46.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:46.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:46.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:46.948 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:47:46.946+0000 7fc88abb9640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:47:47.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:47.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:47.490 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:47:47.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:47:47.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:47.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:47.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:48.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:47:48.938 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:47:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:48.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:49.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:50.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:50.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:50.227 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:47:50.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:47:50.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:50.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:50.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:51.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:51.613 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:47:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:47:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:51.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:51.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:52.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:52.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:52.970 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:47:52.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:47:52.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:52.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:53.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:54.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:54.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:54.260 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:47:54.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:47:54.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:54.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:54.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:55.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:55.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:55.631 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:47:55.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:47:55.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:55.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:55.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:56.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:56.928 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:47:56.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:47:56.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:56.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:57.215 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:58.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:58.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:58.217 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:47:58.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:47:58.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:58.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:47:58.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:47:59.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:47:59.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:47:59.555 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:47:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:47:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:47:59.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:00.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:01.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:01.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:01.182 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:48:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:48:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:01.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:02.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:02.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:02.672 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:48:02.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:48:02.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:02.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:03.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:04.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:04.464 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:48:04.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:48:04.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:04.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:05.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:06.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:06.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:06.292 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:48:06.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:48:06.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:06.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:07.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:48:08.128 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:48:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:09.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:10.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:10.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:10.014 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:48:10.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:48:10.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:10.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:48:10.326 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1500298803,v1:127.0.0.1:6811/1500298803] [v2:127.0.0.1:6812/1500298803,v1:127.0.0.1:6813/1500298803] exists,up ff72e0b6-d830-4cee-9ef7-23bbfd1f394e 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-10T21:48:10.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:48:10.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:48:10.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8f519a1b-3762-49c2-8b56-0ec5353f585d 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8f519a1b-3762-49c2-8b56-0ec5353f585d' 2024-09-10T21:48:10.333 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 8f519a1b-3762-49c2-8b56-0ec5353f585d 2024-09-10T21:48:10.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:48:10.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCavuBmJxDGFBAAvSZRvF5FngdShg9VlEY4uw== 2024-09-10T21:48:10.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCavuBmJxDGFBAAvSZRvF5FngdShg9VlEY4uw=="}' 2024-09-10T21:48:10.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8f519a1b-3762-49c2-8b56-0ec5353f585d -i td/osd-scrub-repair/2/new.json 2024-09-10T21:48:10.680 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:48:10.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:48:10.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCavuBmJxDGFBAAvSZRvF5FngdShg9VlEY4uw== --osd-uuid 8f519a1b-3762-49c2-8b56-0ec5353f585d 2024-09-10T21:48:10.722 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:10.721+0000 7f183c1dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:10.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:10.764+0000 7f183c1dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:10.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:10.765+0000 7f183c1dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:10.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:10.766+0000 7f183c1dd640 -1 bdev(0x563ba8d2dc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:48:10.766 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:10.766+0000 7f183c1dd640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:48:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:48:17.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:48:17.066 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:48:17.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:48:17.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:48:17.426 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:48:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:48:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:48:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:48:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:48:17.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:48:17.454 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:17.453+0000 7fba9cbc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:17.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:17.455+0000 7fba9cbc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:17.457 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:17.456+0000 7fba9cbc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:48:17.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:17.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:48:17.703 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:48:17.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:17.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:18.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:18.594 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:18.594+0000 7fba9cbc3640 -1 Falling back to public interface 2024-09-10T21:48:19.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:19.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:19.009 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:48:19.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:48:19.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:19.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:19.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:20.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:20.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:20.299 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:48:20.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:48:20.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:20.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:20.345 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:20.345+0000 7fba9cbc3640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:48:20.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:21.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:21.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:21.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:48:21.585 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:48:21.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:21.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:21.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:22.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:22.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:22.874 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:48:22.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:48:22.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:22.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:23.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:24.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:24.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:24.243 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:48:24.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:48:24.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:24.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:24.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:25.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:25.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:25.661 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:48:25.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:48:25.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:25.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:25.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:26.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:48:26.950 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:48:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:26.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:27.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:28.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:28.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:28.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:48:28.242 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:48:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:28.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:28.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:29.524 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:48:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:48:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:29.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:30.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:30.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:30.806 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:48:30.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:48:30.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:30.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:31.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:32.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:32.095 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:48:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:48:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:32.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:32.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:33.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:33.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:33.744 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:48:33.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:48:33.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:33.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:34.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:35.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:35.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:35.209 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:48:35.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:48:35.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:35.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:35.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:36.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:36.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:36.495 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:48:36.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:48:36.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:36.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:36.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:37.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:37.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:37.984 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:48:37.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:48:37.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:37.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:38.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:39.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:39.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:39.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:48:39.837 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:48:39.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:39.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:40.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:41.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:41.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:41.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:48:41.263 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:48:41.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:41.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:42.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:43.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:43.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:43.011 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:48:43.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:48:43.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:43.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:43.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:44.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:44.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:44.646 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:48:44.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:48:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:44.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:48:44.964 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3084193744,v1:127.0.0.1:6819/3084193744] [v2:127.0.0.1:6820/3084193744,v1:127.0.0.1:6821/3084193744] exists,up 8f519a1b-3762-49c2-8b56-0ec5353f585d 2024-09-10T21:48:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:48:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:48:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:48:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:48:44.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:48:44.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:48:44.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=98d1090c-4f42-48f6-8558-d44c886f6552 2024-09-10T21:48:44.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 98d1090c-4f42-48f6-8558-d44c886f6552' 2024-09-10T21:48:44.972 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 98d1090c-4f42-48f6-8558-d44c886f6552 2024-09-10T21:48:44.972 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:48:44.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC8vuBm22bKOhAAFW0B8uQMPH//oUuc6ZWS+Q== 2024-09-10T21:48:44.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC8vuBm22bKOhAAFW0B8uQMPH//oUuc6ZWS+Q=="}' 2024-09-10T21:48:44.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 98d1090c-4f42-48f6-8558-d44c886f6552 -i td/osd-scrub-repair/3/new.json 2024-09-10T21:48:45.353 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:48:45.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T21:48:45.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8vuBm22bKOhAAFW0B8uQMPH//oUuc6ZWS+Q== --osd-uuid 98d1090c-4f42-48f6-8558-d44c886f6552 2024-09-10T21:48:45.395 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:45.394+0000 7fbd8d8b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:45.429 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:45.429+0000 7fbd8d8b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:45.430 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:45.430+0000 7fbd8d8b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:45.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:45.430+0000 7fbd8d8b4640 -1 bdev(0x5638b2a6bc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T21:48:45.431 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:45.430+0000 7fbd8d8b4640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T21:48:51.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T21:48:51.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:48:51.825 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T21:48:51.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T21:48:51.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:48:52.191 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T21:48:52.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T21:48:52.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:48:52.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:48:52.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:48:52.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:48:52.219 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:52.218+0000 7f9cfac1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:52.235 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:52.235+0000 7f9cfac1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:52.236 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:52.236+0000 7f9cfac1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:48:52.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T21:48:52.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:48:52.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:48:52.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:48:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:48:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:48:52.502 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:48:52.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:52.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:52.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:53.360 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:53.359+0000 7f9cfac1a640 -1 Falling back to public interface 2024-09-10T21:48:53.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:53.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:53.778 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:48:53.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:48:53.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:53.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:54.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:54.936 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:48:54.936+0000 7f9cfac1a640 -1 osd.3 0 log_to_monitors true 2024-09-10T21:48:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:55.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:48:55.092 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:48:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:55.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:55.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:56.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:56.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:56.432 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:48:56.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:48:56.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:56.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:56.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:57.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:57.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:57.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:48:57.815 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:48:57.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:57.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:58.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:48:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:48:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:48:59.103 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:48:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:48:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:48:59.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:48:59.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:00.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:00.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:00.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:49:00.393 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:49:00.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:00.393 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:00.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:01.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:01.738 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:49:01.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:49:01.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:01.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:02.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:03.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:03.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:03.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:49:03.059 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:49:03.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:03.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:03.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:04.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:04.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:04.350 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:49:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:49:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:04.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:04.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:05.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:05.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:05.699 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:49:05.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:49:05.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:05.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:06.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:07.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:07.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:07.068 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:49:07.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:49:07.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:07.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:07.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:08.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:08.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:08.358 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:49:08.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:49:08.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:08.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:10.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:10.059 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:49:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:49:10.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:10.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:10.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:11.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:11.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:11.857 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:49:11.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:49:11.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:11.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:12.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:13.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:13.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:13.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:49:13.144 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:49:13.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:13.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:13.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:14.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:14.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:14.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:49:14.815 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:49:14.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:14.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:15.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:16.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:16.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:16.531 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:49:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:49:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:16.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:17.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:18.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:18.312 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:49:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:49:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:18.312 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:18.601 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1102787615,v1:127.0.0.1:6827/1102787615] [v2:127.0.0.1:6828/1102787615,v1:127.0.0.1:6829/1102787615] exists,up 98d1090c-4f42-48f6-8558-d44c886f6552 2024-09-10T21:49:18.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:49:18.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:49:18.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:49:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-10T21:49:18.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:49:18.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:49:19.658 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:49:19.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:49:20.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-10T21:49:20.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:49:20.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:49:20.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:49:20.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:49:20.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:49:20.673 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:49:20.673 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:49:20.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:49:20.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:49:20.673 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:49:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:49:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:49:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:49:20.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:49:20.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:49:20.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:49:21.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:21.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:49:21.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-10T21:49:21.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-10T21:49:21.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-10T21:49:21.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:21.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:49:21.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-10T21:49:21.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-10T21:49:21.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-09-10T21:49:21.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:21.364 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:49:21.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-10T21:49:21.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-10T21:49:21.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509449' 2024-09-10T21:49:21.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:21.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:49:21.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-10T21:49:21.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-10T21:49:21.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509449 3-85899345926' 2024-09-10T21:49:21.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:21.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-10T21:49:21.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:21.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:49:21.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-10T21:49:21.572 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:21.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-10T21:49:21.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-10T21:49:21.574 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836505 2024-09-10T21:49:21.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:21.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-10T21:49:21.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:49:22.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:49:22.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-10T21:49:23.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:23.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-10T21:49:23.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:23.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:49:23.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-10T21:49:23.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:23.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-10T21:49:23.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-10T21:49:23.138 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672978 2024-09-10T21:49:23.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:49:23.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-10T21:49:23.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:23.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-10T21:49:23.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:23.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:49:23.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-10T21:49:23.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:23.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-10T21:49:23.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-10T21:49:23.416 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509449 2024-09-10T21:49:23.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:49:23.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-10T21:49:23.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:23.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-10T21:49:23.694 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:23.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:49:23.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-10T21:49:23.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:23.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-10T21:49:23.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-10T21:49:23.697 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345926 2024-09-10T21:49:23.697 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:49:23.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-10T21:49:23.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:49:23.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:23.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:24.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:49:24.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:49:24.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:49:24.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:49:24.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:49:24.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:24.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:49:24.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:49:24.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:49:24.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:49:25.367 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:49:25.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:49:25.752 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:49:25.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:49:25.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:49:25.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:49:26.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T21:49:26.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T21:49:26.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:49:26.166 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:49:26.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:49:26.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-10T21:49:26.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T21:49:26.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:49:26.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-10T21:49:26.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T21:49:26.525 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:49:26.525 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-10T21:49:26.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 438560"' 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 438560' 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 438561"' 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 438561' 2024-09-10T21:49:26.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/438563: /' 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 438565"' 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 438565' 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 438560 438561 438565' 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 438560 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/438566: /' 2024-09-10T21:49:26.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:49:26.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/438569: /' 2024-09-10T21:49:29.707 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:49:29.707 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:49:29.707 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:49:29.708 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:49:29.709 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:49:29.710 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:49:29.710 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:49:29.710 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:29.711 INFO:tasks.workunit.client.0.smithi181.stderr:438569: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:49:29.712 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:49:29.713 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:49:29.713 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:49:29.713 INFO:tasks.workunit.client.0.smithi181.stderr:438569: start osd.3 2024-09-10T21:49:29.713 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:49:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:4438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:49:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:49:30.178 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:49:30.179 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:49:30.180 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:49:30.181 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:49:30.182 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:49:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:49:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:49:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:49:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:438563: start osd.0 2024-09-10T21:49:30.183 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:49:30.219 INFO:tasks.workunit.client.0.smithi181.stderr:4438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.219 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:49:30.219 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:49:30.219 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:49:30.219 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.220 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:49:30.221 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:49:30.222 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:49:30.223 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:49:30.224 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:49:30.224 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:49:30.224 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:49:30.224 INFO:tasks.workunit.client.0.smithi181.stderr:438566: start osd.2 2024-09-10T21:49:30.224 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:49:34.155 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:49:34.155 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2024-09-10T21:49:29.739+0000 7f04d1e9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2024-09-10T21:49:29.741+0000 7f04d1e9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2024-09-10T21:49:29.742+0000 7f04d1e9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.156 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 0 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 1 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2024-09-10T21:49:31.388+0000 7f04d1e9c640 -1 Falling back to public interface 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.157 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 3 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:34.158 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2024-09-10T21:49:33.882+0000 7f04d1e9c640 -1 osd.3 27 log_to_monitors true 2024-09-10T21:49:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers38566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:49:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:49:34.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2024-09-10T21:49:30.249+0000 7f036e7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2024-09-10T21:49:30.251+0000 7f036e7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2024-09-10T21:49:30.252+0000 7f036e7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 0 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2024-09-10T21:49:31.397+0000 7f036e7c5640 -1 Falling back to public interface 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 1 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2024-09-10T21:49:33.815+0000 7f036e7c5640 -1 osd.2 27 log_to_monitors true 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 3 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438566: osd.2 up in weight 1 up_from 32 up_thru 0 down_at 28 last_cl38563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:49:34.788 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2024-09-10T21:49:30.206+0000 7f1f3d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2024-09-10T21:49:30.208+0000 7f1f3d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2024-09-10T21:49:30.210+0000 7f1f3d20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:49:34.789 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 0 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2024-09-10T21:49:31.567+0000 7f1f3d20c640 -1 Falling back to public interface 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 1 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.790 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2024-09-10T21:49:34.145+0000 7f1f3d20c640 -1 osd.0 27 log_to_monitors true 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 3 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:34.791 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:35.598 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersean_interval [15,27) [v2:127.0.0.1:6818/1103572498,v1:127.0.0.1:6819/1103572498] [v2:127.0.0.1:6820/1103572498,v1:127.0.0.1:6821/1103572498] exists,up 8f519a1b-3762-49c2-8b56-0ec5353f585d 2024-09-10T21:49:35.598 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:49:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 1 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 2 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: 3' 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-10T21:49:35.600 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920770' 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920770 1-42949672982' 2024-09-10T21:49:35.601 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph t.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 4 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: osd.3 up in weight 1 up_from 32 up_thru 0 down_at 28 last_clean_interval [20,27) [v2:127.0.0.1:6802/75501441,v1:127.0.0.1:6803/75501441] [v2:127.0.0.1:6804/75501441,v1:127.0.0.1:6805/75501441] exists,up 98d1090c-4f42-48f6-8558-d44c886f6552 2024-09-10T21:49:36.227 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:49:36.228 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:49:36.229 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 1 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 2 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: 3' 2024-09-10T21:49:36.230 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920771' 2024-09-10T21:49:36.231 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:36.692 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:36.692 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 4 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: osd.0 up in weight 1 up_from 33 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6826/707225937,v1:127.0.0.1:6827/707225937] [v2:127.0.0.1:6828/707225937,v1:127.0.0.1:6829/707225937] exists,up bdf0285d-2f61-4ff2-a58f-0933e9a52792 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:49:36.693 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:49:36.694 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 1 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 2 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: 3' 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920772' 2024-09-10T21:49:36.695 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpersell osd.2 flush_pg_stats 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920770 1-42949672982 2-137438953474' 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-10T21:49:37.784 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920770 1-42949672982 2-137438953474 3-137438953474' 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-141733920770 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-141733920770 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 141733920770' 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: waiting osd.0 seq 141733920770 2024-09-10T21:49:37.785 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 141733920770 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920770 -lt 141733920770 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:37.786 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T21:49:37.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T21:49:37.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: waiting osd.1 seq 42949672982 2024-09-10T21:49:37.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:49:37.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-10T21:49:37.787 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:38.164 INFO:tasks.workunit.client.0.smithi181.stderr:4385:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:49:38.164 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T21:49:38.164 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T21:49:38.164 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920771 1-42949672983' 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920771 1-42949672983 2-137438953475' 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-10T21:49:38.165 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920771 1-42949672983 2-137438953475 3-137438953475' 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-141733920771 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-141733920771 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: waiting osd.0 seq 141733920771 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920770 -lt 141733920771 2024-09-10T21:49:38.166 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T21:49:38.167 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubu.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920772 1-42949672984' 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953476 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953476 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920772 1-42949672984 2-137438953476' 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:49:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953476 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953476 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-141733920772 1-42949672984 2-137438953476 3-137438953476' 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-141733920772 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-141733920772 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 141733920772' 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: waiting osd.0 seq 141733920772 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920770 -lt 141733920772 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:49:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-10T21:49:38.631 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/66: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-137438953474 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-137438953474 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 137438953474' 2024-09-10T21:49:39.108 INFO:tasks.workunit.client.0.smithi181.stderr:438566: waiting osd.2 seq 137438953474 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953474 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-137438953474 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-137438953474 2024-09-10T21:49:39.109 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 137438953474' 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: waiting osd.3 seq 137438953474 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953474 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:39.110 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:49:39.111 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:49:39.524 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438569: waiting osd.1 seq 42949672983 2024-09-10T21:49:39.525 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672983 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-137438953475 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-137438953475 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: waiting osd.2 seq 137438953475 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953475 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:39.526 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-137438953475 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-137438953475 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 137438953475' 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: waiting osd.3 seq 137438953475 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953475 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:39.527 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:49:39.528 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:49:39.528 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:49:39.528 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:49:39.528 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:49:39.884 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale")ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-10T21:49:39.884 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: waiting osd.1 seq 42949672984 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-137438953476 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-137438953476 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953476 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 137438953476' 2024-09-10T21:49:39.885 INFO:tasks.workunit.client.0.smithi181.stderr:438563: waiting osd.2 seq 137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 137438953476' 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: waiting osd.3 seq 137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953476 2024-09-10T21:49:39.886 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:49:39.887 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stal | not)' 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:49:40.168 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:49:40.169 INFO:tasks.workunit.client.0.smithi181.stderr:438569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:49:40.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:49:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:e") | not)' 2024-09-10T21:49:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:49:40.508 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:438563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 438561 2024-09-10T21:49:40.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 438565 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:49:40.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:40.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:40.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:40.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:41.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:41.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T21:49:41.320 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:41.333 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:41.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:41.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:41.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:49:19.490682+0000 '>' 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:41.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:49:42.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:49:42.605 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:49:42.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:49:42.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:42.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:42.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:42.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:42.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:49:19.490682+0000 '>' 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:42.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:49:43.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:49:43.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:49:43.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:49:43.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:43.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:43.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:43.890 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:49:19.490682+0000 '>' 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:44.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:49:45.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:49:45.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:49:45.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:49:45.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:45.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:45.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:45.213 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:45.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:49:19.490682+0000 '>' 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:45.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:49:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:49:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:49:46.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:49:46.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:49:46.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:49:46.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:49:46.496 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:49:46.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:49:41.543343+0000 '>' 2024-09-10T21:49:19.490682+0000 2024-09-10T21:49:46.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:49:46.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:49:48.772 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-10T21:49:48.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-10T21:49:48.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-10T21:49:48.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-10T21:49:49.123 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:49:49.123 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:49:49.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-10T21:49:49.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-10T21:49:49.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-10T21:49:49.466 INFO:tasks.workunit.client.0.smithi181.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 64s) 2024-09-10T21:49:49.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-10T21:49:49.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-10T21:49:49.892 INFO:tasks.workunit.client.0.smithi181.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 64s) 2024-09-10T21:49:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-10T21:49:49.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-10T21:49:50.235 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:49:50.236 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:49:50.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:49:50.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:49:50.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:49:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:49:50.347 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:49:50.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:49:50.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:49:50.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:49:50.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:49:50.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:49:50.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:49:50.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:49:50.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:49:50.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:49:50.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:49:50.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:49:50.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:50.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:50.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:49:50.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:49:50.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:49:50.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:49:50.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:49:50.377 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:49:50.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:49:50.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:49:50.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:49:50.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:49:50.379 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:49:50.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:49:50.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:49:50.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:49:50.381 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:49:50.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:49:50.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:49:50.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:49:50.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:49:50.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:50.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:50.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:49:50.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:49:50.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:49:50.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-10T21:49:50.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:49:50.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:50.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:50.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52449 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-09-10T21:49:50.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-10T21:49:50.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-10T21:49:50.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:49:50.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:50.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:50.903 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:50.904 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:50.904 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:50.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:50.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T21:49:51.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:49:51.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:49:51.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:51.110 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:49:51.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:49:51.111 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get fsid 2024-09-10T21:49:51.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:49:51.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:49:51.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:49:51.188 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:49:51.189 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:49:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:49:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:49:51.189 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52449/ceph-mon.a.asok 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:49:51.190 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52449/ceph-mon.a.asok config get mon_host 2024-09-10T21:49:51.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-10T21:49:51.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:49:52.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:52.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:49:52.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:49:52.102 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:49:52.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:49:52.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:49:52.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:49:52.109 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:49:52.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2f1d17c5-cc41-4862-a757-a9b1a7813a8d 2024-09-10T21:49:52.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2f1d17c5-cc41-4862-a757-a9b1a7813a8d' 2024-09-10T21:49:52.109 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 2f1d17c5-cc41-4862-a757-a9b1a7813a8d 2024-09-10T21:49:52.110 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:49:52.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAv+Bmwax7BxAAyWWQCxfk2W8fwtY7LU8NtA== 2024-09-10T21:49:52.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAv+Bmwax7BxAAyWWQCxfk2W8fwtY7LU8NtA=="}' 2024-09-10T21:49:52.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2f1d17c5-cc41-4862-a757-a9b1a7813a8d -i td/osd-scrub-repair/0/new.json 2024-09-10T21:49:52.328 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:49:52.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-10T21:49:52.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAv+Bmwax7BxAAyWWQCxfk2W8fwtY7LU8NtA== --osd-uuid 2f1d17c5-cc41-4862-a757-a9b1a7813a8d 2024-09-10T21:49:52.368 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:52.368+0000 7fa074efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:52.379 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:52.378+0000 7fa074efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:52.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:52.379+0000 7fa074efe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:52.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:52.380+0000 7fa074efe640 -1 bdev(0x561af997bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:49:52.380 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:52.380+0000 7fa074efe640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-10T21:49:58.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-10T21:49:58.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:49:58.012 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:49:58.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:49:58.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:49:58.371 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:49:58.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:49:58.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:49:58.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:49:58.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:49:58.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:49:58.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:58.398+0000 7f67defdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:58.401 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:58.401+0000 7f67defdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:58.402 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:58.402+0000 7f67defdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:49:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:58.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:49:58.647 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:49:58.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:58.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:49:58.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:49:59.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:49:59.767+0000 7f67defdd640 -1 Falling back to public interface 2024-09-10T21:49:59.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:49:59.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:49:59.924 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:49:59.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:49:59.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:49:59.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:00.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:01.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:01.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:01.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:50:01.212 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:50:01.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:01.212 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:01.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:01.726 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:01.725+0000 7f67defdd640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:50:02.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:50:02.492 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:50:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:02.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:02.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:03.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:03.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:03.903 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:50:03.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:50:03.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:03.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:04.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:05.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:05.192 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:50:05.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:05.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:50:05.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:05.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:06.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:06.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:06.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:50:06.503 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:50:06.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:06.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:06.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:07.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:07.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:07.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:50:07.792 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:50:07.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:07.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:08.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:09.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:09.082 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:50:09.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:09.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:50:09.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:09.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:09.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:10.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:10.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:10.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:50:10.373 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:50:10.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:10.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:10.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:11.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:11.734 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:50:11.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:11.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:50:11.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:11.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:12.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:13.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:13.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:13.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:50:13.022 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:50:13.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:13.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:13.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:14.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:50:14.401 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:50:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:14.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:14.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:15.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:15.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:15.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:50:15.820 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:50:15.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:15.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:16.604 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:17.606 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:50:17.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:50:17.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:17.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:18.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:19.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:19.042 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:50:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:50:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:19.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:19.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:20.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:20.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:20.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:50:20.637 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:50:20.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:20.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:20.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:21.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:21.928 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:50:21.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:21.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:50:21.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:21.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:22.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:23.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:23.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:23.431 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:50:23.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:50:23.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:23.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:23.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:24.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:24.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:24.717 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:50:24.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:50:24.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:24.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:50:25.002 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1757393305,v1:127.0.0.1:6803/1757393305] [v2:127.0.0.1:6804/1757393305,v1:127.0.0.1:6805/1757393305] exists,up 2f1d17c5-cc41-4862-a757-a9b1a7813a8d 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-10T21:50:25.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:50:25.007 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:50:25.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:50:25.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-10T21:50:25.009 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:50:25.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d68bfe50-ec29-40a1-a67a-d40904ba66e4 2024-09-10T21:50:25.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d68bfe50-ec29-40a1-a67a-d40904ba66e4' 2024-09-10T21:50:25.009 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 d68bfe50-ec29-40a1-a67a-d40904ba66e4 2024-09-10T21:50:25.010 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:50:25.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhv+BmMmh0ARAALBvXiHHPdoeB9r97Glvh7w== 2024-09-10T21:50:25.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhv+BmMmh0ARAALBvXiHHPdoeB9r97Glvh7w=="}' 2024-09-10T21:50:25.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d68bfe50-ec29-40a1-a67a-d40904ba66e4 -i td/osd-scrub-repair/1/new.json 2024-09-10T21:50:25.368 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:50:25.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-10T21:50:25.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhv+BmMmh0ARAALBvXiHHPdoeB9r97Glvh7w== --osd-uuid d68bfe50-ec29-40a1-a67a-d40904ba66e4 2024-09-10T21:50:25.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:25.408+0000 7f5f3ba15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:25.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:25.543+0000 7f5f3ba15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:25.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:25.545+0000 7f5f3ba15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:25.545 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:25.545+0000 7f5f3ba15640 -1 bdev(0x55c41b609c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:50:25.546 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:25.545+0000 7f5f3ba15640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-10T21:50:31.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-10T21:50:31.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:50:31.331 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:50:31.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:50:31.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:50:31.690 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:50:31.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:50:31.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:50:31.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:50:31.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:50:31.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:50:31.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:31.717+0000 7f9d7e612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:31.720 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:31.719+0000 7f9d7e612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:31.721 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:31.720+0000 7f9d7e612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:31.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:50:31.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:50:31.969 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:50:31.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:31.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:32.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:33.086 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:33.085+0000 7f9d7e612640 -1 Falling back to public interface 2024-09-10T21:50:33.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:33.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:33.264 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:50:33.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:50:33.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:33.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:33.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:34.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:34.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:34.554 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:50:34.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:50:34.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:34.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:34.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:34.974 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:34.974+0000 7f9d7e612640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:50:35.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:35.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:35.844 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:50:35.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:50:35.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:35.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:36.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:37.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:37.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:37.244 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:50:37.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:50:37.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:37.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:37.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:38.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:38.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:38.531 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:50:38.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:50:38.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:38.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:38.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:39.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:39.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:39.858 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:50:39.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:50:39.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:39.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:40.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:41.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:41.151 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:50:41.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:50:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:41.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:41.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:42.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:42.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:42.438 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:50:42.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:50:42.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:42.439 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:42.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:43.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:50:43.727 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:50:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:43.727 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:44.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:45.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:45.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:45.115 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:50:45.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:50:45.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:45.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:45.406 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:46.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:46.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:46.408 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:50:46.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:50:46.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:46.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:46.796 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:47.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:47.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:47.797 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:50:47.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:50:47.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:47.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:48.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:49.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:49.328 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:50:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:50:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:49.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:49.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:50.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:50.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:50.885 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:50:50.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:50:50.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:50.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:51.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:50:52.334 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:50:52.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:52.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:53.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:54.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:54.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:54.003 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:50:54.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:50:54.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:54.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:54.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:55.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:55.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:55.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:50:55.388 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:50:55.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:55.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:55.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:56.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:56.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:56.880 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:50:56.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:50:56.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:56.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:57.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:50:58.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:50:58.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:50:58.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:50:58.210 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:50:58.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:50:58.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:50:58.496 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2465586124,v1:127.0.0.1:6811/2465586124] [v2:127.0.0.1:6812/2465586124,v1:127.0.0.1:6813/2465586124] exists,up d68bfe50-ec29-40a1-a67a-d40904ba66e4 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:50:58.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:50:58.501 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:50:58.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c30ccc1-9850-4a55-b5f8-32600ef9d76a 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7c30ccc1-9850-4a55-b5f8-32600ef9d76a' 2024-09-10T21:50:58.503 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 7c30ccc1-9850-4a55-b5f8-32600ef9d76a 2024-09-10T21:50:58.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:50:58.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCv+BmDwrjHhAAqie9PuhwfCdQQKhIdmbpSA== 2024-09-10T21:50:58.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCv+BmDwrjHhAAqie9PuhwfCdQQKhIdmbpSA=="}' 2024-09-10T21:50:58.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c30ccc1-9850-4a55-b5f8-32600ef9d76a -i td/osd-scrub-repair/2/new.json 2024-09-10T21:50:58.877 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:50:58.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-10T21:50:58.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCv+BmDwrjHhAAqie9PuhwfCdQQKhIdmbpSA== --osd-uuid 7c30ccc1-9850-4a55-b5f8-32600ef9d76a 2024-09-10T21:50:58.920 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:58.919+0000 7f906caeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:58.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:58.965+0000 7f906caeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:58.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:58.966+0000 7f906caeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:50:58.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:58.966+0000 7f906caeb640 -1 bdev(0x55fd3986fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-10T21:50:58.967 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:50:58.966+0000 7f906caeb640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-10T21:51:05.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-10T21:51:05.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:51:05.037 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T21:51:05.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T21:51:05.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:51:05.395 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T21:51:05.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T21:51:05.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:51:05.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:51:05.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:51:05.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:51:05.423 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:05.421+0000 7ff85345e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:05.424 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:05.423+0000 7ff85345e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:05.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:05.424+0000 7ff85345e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:05.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T21:51:05.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:51:05.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:51:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:51:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:51:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:05.674 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:51:05.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:51:05.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:05.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:05.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:06.794 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:06.793+0000 7ff85345e640 -1 Falling back to public interface 2024-09-10T21:51:06.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:06.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:06.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:51:06.954 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:51:06.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:06.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:07.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:08.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:08.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:08.239 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:51:08.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:51:08.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:08.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:08.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:08.810 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:08.809+0000 7ff85345e640 -1 osd.2 0 log_to_monitors true 2024-09-10T21:51:09.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:09.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:09.609 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:51:09.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:51:09.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:09.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:09.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:10.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:10.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:10.894 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:51:10.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:51:10.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:10.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:11.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:12.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:12.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:12.186 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:51:12.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:51:12.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:12.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:12.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:13.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:13.473 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:51:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:51:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:13.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:13.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:14.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:14.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:14.825 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:51:14.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:51:14.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:14.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:15.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:16.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:16.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:16.115 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:51:16.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:51:16.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:16.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:16.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:17.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:17.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:17.467 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:51:17.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:51:17.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:17.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:17.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:18.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:18.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:18.832 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:51:18.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:51:18.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:18.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:19.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:20.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:51:20.122 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:51:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:20.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:20.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:21.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:21.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:21.613 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:51:21.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:51:21.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:21.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:22.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:23.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:51:23.116 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:51:23.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:23.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:23.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:24.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:24.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:24.654 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:51:24.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:51:24.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:24.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:25.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:26.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:26.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:26.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:51:26.064 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:51:26.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:26.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:26.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:27.449 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:51:27.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:51:27.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:27.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:28.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:29.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:29.078 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:51:29.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:51:29.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:29.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:29.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:30.883 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:30.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:30.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:51:30.884 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:51:30.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:30.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:31.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:32.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:32.397 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:32.398 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:51:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:51:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:32.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:51:32.682 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/573723945,v1:127.0.0.1:6819/573723945] [v2:127.0.0.1:6820/573723945,v1:127.0.0.1:6821/573723945] exists,up 7c30ccc1-9850-4a55-b5f8-32600ef9d76a 2024-09-10T21:51:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:51:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:51:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:51:32.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:51:32.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:51:32.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:51:32.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:51:32.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1bdf0596-69d5-4b31-8b18-81740eea1f65 2024-09-10T21:51:32.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1bdf0596-69d5-4b31-8b18-81740eea1f65' 2024-09-10T21:51:32.690 INFO:tasks.workunit.client.0.smithi181.stdout:add osd3 1bdf0596-69d5-4b31-8b18-81740eea1f65 2024-09-10T21:51:32.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:51:32.705 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkv+BmBVUFKhAAyQkrYOn4DzioyZImGNyxRQ== 2024-09-10T21:51:32.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkv+BmBVUFKhAAyQkrYOn4DzioyZImGNyxRQ=="}' 2024-09-10T21:51:32.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1bdf0596-69d5-4b31-8b18-81740eea1f65 -i td/osd-scrub-repair/3/new.json 2024-09-10T21:51:33.089 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:51:33.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-10T21:51:33.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkv+BmBVUFKhAAyQkrYOn4DzioyZImGNyxRQ== --osd-uuid 1bdf0596-69d5-4b31-8b18-81740eea1f65 2024-09-10T21:51:33.131 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:33.131+0000 7f9ca8c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:33.174 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:33.173+0000 7f9ca8c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:33.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:33.174+0000 7f9ca8c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:33.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:33.175+0000 7f9ca8c2e640 -1 bdev(0x55f2d3853c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-10T21:51:33.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:33.175+0000 7f9ca8c2e640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-10T21:51:39.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-10T21:51:39.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:51:39.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T21:51:39.638 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd3 key to auth repository 2024-09-10T21:51:39.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:51:39.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T21:51:39.998 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.3 2024-09-10T21:51:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:51:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:51:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:51:39.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:51:40.026 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:40.025+0000 7f9e95487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:40.027+0000 7f9e95487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:40.029 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:40.028+0000 7f9e95487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:40.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:51:40.309 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:51:40.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:40.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:40.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:41.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:41.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:41.588 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:51:41.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:51:41.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:41.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:41.875 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:41.874+0000 7f9e95487640 -1 Falling back to public interface 2024-09-10T21:51:41.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:42.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:42.877 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:51:42.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:51:42.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:42.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:43.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:43.548 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:51:43.547+0000 7f9e95487640 -1 osd.3 0 log_to_monitors true 2024-09-10T21:51:44.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:44.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:44.168 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:51:44.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:51:44.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:44.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:44.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:45.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:45.593 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:51:45.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:51:45.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:45.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:45.878 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:46.880 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:51:46.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:51:46.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:46.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:47.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:48.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:48.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:48.168 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:51:48.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:51:48.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:48.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:48.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:49.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:49.495 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:51:49.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:51:49.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:49.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:49.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:50.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:50.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:50.785 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:51:50.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:51:50.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:50.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:51.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:52.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:52.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:52.075 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:51:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:51:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:52.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:52.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:53.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:53.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:53.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:51:53.453 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:51:53.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:53.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:53.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:54.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:54.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:54.741 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:51:54.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:51:54.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:54.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:55.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:56.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:56.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:56.029 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:51:56.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:51:56.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:56.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:56.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:57.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:57.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:57.623 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:51:57.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:51:57.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:57.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:58.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:51:59.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:51:59.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:51:59.042 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:51:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:51:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:51:59.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:51:59.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:00.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:00.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:00.668 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:52:00.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:52:00.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:00.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:01.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:02.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:02.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:02.264 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:52:02.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:52:02.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:02.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:02.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:03.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:03.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:03.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:52:03.633 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:52:03.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:03.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:04.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:05.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:05.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:05.127 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:52:05.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:52:05.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:05.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:05.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:06.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:06.875 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:52:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:52:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:06.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:07.186 INFO:tasks.workunit.client.0.smithi181.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2377602039,v1:127.0.0.1:6827/2377602039] [v2:127.0.0.1:6828/2377602039,v1:127.0.0.1:6829/2377602039] exists,up 1bdf0596-69d5-4b31-8b18-81740eea1f65 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T21:52:07.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-10T21:52:07.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-10T21:52:07.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-10T21:52:08.250 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'ecpool' created 2024-09-10T21:52:08.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:52:09.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T21:52:09.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-10T21:52:09.848 INFO:tasks.workunit.client.0.smithi181.stderr:set pool 1 allow_ec_overwrites to true 2024-09-10T21:52:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T21:52:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:52:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:52:09.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:52:09.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:52:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:52:09.861 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:52:09.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:52:09.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:52:09.862 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:52:09.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:52:09.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:52:09.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:52:09.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:52:09.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:52:09.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:52:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:10.481 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:52:10.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-10T21:52:10.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-10T21:52:10.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-10T21:52:10.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:10.699 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:52:10.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-10T21:52:10.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-10T21:52:10.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-09-10T21:52:10.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:10.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:52:10.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-10T21:52:10.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-10T21:52:10.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509453' 2024-09-10T21:52:10.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:10.908 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:52:11.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345925 2024-09-10T21:52:11.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345925 2024-09-10T21:52:11.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509453 3-85899345925' 2024-09-10T21:52:11.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:11.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-10T21:52:11.011 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:11.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:52:11.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-10T21:52:11.012 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:11.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-10T21:52:11.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-10T21:52:11.013 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836505 2024-09-10T21:52:11.014 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:11.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-10T21:52:11.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:52:12.291 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:52:12.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:12.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-10T21:52:12.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:12.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-10T21:52:12.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:12.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:52:12.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-10T21:52:12.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:12.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-10T21:52:12.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-10T21:52:12.579 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672977 2024-09-10T21:52:12.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:52:12.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-10T21:52:12.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:12.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-10T21:52:12.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:12.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:52:12.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-10T21:52:12.856 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:12.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-10T21:52:12.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-10T21:52:12.857 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509453 2024-09-10T21:52:12.858 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:52:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-10T21:52:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:13.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345925 2024-09-10T21:52:13.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:13.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:52:13.135 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345925 2024-09-10T21:52:13.136 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345925 2024-09-10T21:52:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-09-10T21:52:13.137 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.3 seq 85899345925 2024-09-10T21:52:13.137 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:52:13.416 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345925 -lt 85899345925 2024-09-10T21:52:13.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:52:13.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:13.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:13.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:52:13.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:52:13.760 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:52:13.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:52:13.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:52:13.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:52:13.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:52:13.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:52:14.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:52:14.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:52:14.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:14.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:14.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-10T21:52:14.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-10T21:52:14.760 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:52:14.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-10T21:52:15.162 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:52:15.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-10T21:52:15.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-10T21:52:15.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-10T21:52:15.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-10T21:52:15.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-10T21:52:15.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-10T21:52:15.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:52:15.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:52:15.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-10T21:52:15.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-10T21:52:15.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-10T21:52:15.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T21:52:15.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T21:52:15.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:52:15.944 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T21:52:16.292 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:0 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:2 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:3' 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:52:16.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 450803"' 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 450803' 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 450804"' 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 450804' 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/450806: /' 2024-09-10T21:52:16.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 450808"' 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 450808' 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/450809: /' 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 450803 450804 450808' 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 450803 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:52:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/450812: /' 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.475 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-10T21:52:19.476 INFO:tasks.workunit.client.0.smithi181.stderr:450812: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:52:19.477 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:52:19.479 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-10T21:52:19.480 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T21:52:19.481 INFO:tasks.workunit.client.0.smithi181.stderr:450812: start osd.3 2024-09-10T21:52:19.481 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:52:19.559 INFO:tasks.workunit.client.0.smithi181.stderr:4450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.559 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:52:19.559 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:19.560 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:52:19.561 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:52:19.562 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-10T21:52:19.562 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:52:19.562 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:19.563 INFO:tasks.workunit.client.0.smithi181.stderr:450806: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:52:19.564 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: start osd.0 2024-09-10T21:52:19.565 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:52:19.700 INFO:tasks.workunit.client.0.smithi181.stderr:4450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.700 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T21:52:19.700 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T21:52:19.700 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:19.701 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-10T21:52:19.702 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-10T21:52:19.703 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:52:19.704 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: start osd.2 2024-09-10T21:52:19.705 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3ef2afa-1c77-4617-8afa-c8101aa3ef65 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52449/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2024-09-10T21:52:19.730+0000 7f9058418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2024-09-10T21:52:19.733+0000 7f9058418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2024-09-10T21:52:19.734+0000 7f9058418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:52:24.263 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 0 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2024-09-10T21:52:21.325+0000 7f9058418640 -1 Falling back to public interface 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.264 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 1 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2024-09-10T21:52:23.462+0000 7f9058418640 -1 osd.2 29 log_to_monitors true 2024-09-10T21:52:24.265 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.266 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.266 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:52:24.266 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 3 2024-09-10T21:52:24.266 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.266 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:52:24.267 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers50812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-10T21:52:24.267 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T21:52:24.267 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2024-09-10T21:52:19.509+0000 7fb3e8c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2024-09-10T21:52:19.511+0000 7fb3e8c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2024-09-10T21:52:19.512+0000 7fb3e8c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.268 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 0 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2024-09-10T21:52:20.874+0000 7fb3e8c14640 -1 Falling back to public interface 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 1 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.269 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2024-09-10T21:52:23.269+0000 7fb3e8c14640 -1 osd.3 29 log_to_monitors true 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 3 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.270 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers50806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2024-09-10T21:52:19.604+0000 7fce06c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2024-09-10T21:52:19.607+0000 7fce06c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2024-09-10T21:52:19.608+0000 7fce06c33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:52:24.271 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 0 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 1 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2024-09-10T21:52:21.205+0000 7fce06c33640 -1 Falling back to public interface 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.272 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 3 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:24.273 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2024-09-10T21:52:23.944+0000 7fce06c33640 -1 osd.0 29 log_to_monitors true 2024-09-10T21:52:26.149 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:26.149 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 4 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: osd.3 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [20,29) [v2:127.0.0.1:6802/115362284,v1:127.0.0.1:6803/115362284] [v2:127.0.0.1:6804/115362284,v1:127.0.0.1:6805/115362284] exists,up 1bdf0596-69d5-4b31-8b18-81740eea1f65 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:52:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:52:26.151 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:52:26.152 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:52:26.152 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:52:26.152 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:52:26.152 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:52:26.152 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 1 2024-09-10T21:52:26.153 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 2 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: 3' 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822658 2024-09-10T21:52:26.154 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822658 2024-09-10T21:52:26.155 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658' 2024-09-10T21:52:26.155 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:26.158 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:26.158 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:26.158 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:26.158 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:52:26.158 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 4 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: osd.2 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6826/356279024,v1:127.0.0.1:6827/356279024] [v2:127.0.0.1:6828/356279024,v1:127.0.0.1:6829/356279024] exists,up 7c30ccc1-9850-4a55-b5f8-32600ef9d76a 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:52:26.159 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:52:26.160 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:52:26.160 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:52:26.160 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:52:26.160 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:52:26.160 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:52:26.161 INFO:tasks.workunit.client.0.smithi181.stderr:450809: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:52:26.161 INFO:tasks.workunit.client.0.smithi181.stderr:450809: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:52:26.161 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:52:26.161 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:52:26.161 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:52:26.162 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:52:26.162 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:52:26.162 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:52:26.162 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:52:26.162 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 1 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 2 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: 3' 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:52:26.163 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:26.164 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:52:26.164 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-10T21:52:26.164 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-10T21:52:26.164 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659' 2024-09-10T21:52:26.164 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:26.165 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:26.165 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:26.165 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 4 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: osd.0 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6818/2533975533,v1:127.0.0.1:6819/2533975533] [v2:127.0.0.1:6820/2533975533,v1:127.0.0.1:6821/2533975533] exists,up 2f1d17c5-cc41-4862-a757-a9b1a7813a8d 2024-09-10T21:52:26.166 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:52:26.167 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:52:26.167 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:52:26.167 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T21:52:26.167 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:52:26.167 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:52:26.168 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:52:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:52:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:52:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:52:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:52:26.169 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 1 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 2 2024-09-10T21:52:26.170 INFO:tasks.workunit.client.0.smithi181.stderr:450806: 3' 2024-09-10T21:52:26.171 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:52:26.171 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:26.171 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:52:26.171 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-10T21:52:26.171 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-10T21:52:26.172 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-10T21:52:26.172 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:28.112 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:52:28.112 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T21:52:28.112 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T21:52:28.112 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672982' 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855363 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855363 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672982 2-150323855363' 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855363 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855363 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672982 2-150323855363 3-150323855363' 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822659 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822659 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: waiting osd.0 seq 154618822659 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822659 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:28.114 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822659 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T21:52:28.115 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/ceppers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:52:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T21:52:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T21:52:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 1-42949672983' 2024-09-10T21:52:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 1-42949672983 2-150323855364' 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 1-42949672983 2-150323855364 3-150323855364' 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:52:28.128 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: waiting osd.0 seq 154618822660 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822660 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:28.129 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:52:28.130 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T21:52:28.130 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/htest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: waiting osd.1 seq 42949672982 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855363 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:29.517 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855363 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855363 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: waiting osd.2 seq 150323855363 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855363 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:29.518 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855363 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855363 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: waiting osd.3 seq 150323855363 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:29.519 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:52:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:52:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:52:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:52:29.520 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: waiting osd.1 seq 42949672983 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855364 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:29.522 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855364 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: waiting osd.2 seq 150323855364 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855364 2024-09-10T21:52:29.523 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855364 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855364' 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: waiting osd.3 seq 150323855364 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:29.524 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | ns.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672981' 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T21:52:29.525 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672981 2-150323855362' 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672981 2-150323855362 3-150323855362' 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822658 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:52:29.526 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822658 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822658 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822658' 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: waiting osd.0 seq 154618822658 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822658 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822658 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:52:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T21:52:29.528 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:52:29.528 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822658 2024-09-10T21:52:29.528 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:29.528 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:4508' 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:52:30.172 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:450809: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:52:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:52:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:450809: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:52:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:ot)' 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:450806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:52:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 450804 2024-09-10T21:52:30.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-10T21:52:30.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 450808 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:12: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-10T21:52:30.812 INFO:tasks.workunit.client.0.smithi181.stderr:450812: waiting osd.1 seq 42949672981 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672981 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855362 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855362 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855362' 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: waiting osd.2 seq 150323855362 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855362 2024-09-10T21:52:30.813 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855362 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855362 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855362' 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: waiting osd.3 seq 150323855362 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855362 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:30.814 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:30.815 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:52:30.815 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:52:31.434 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:52:31.435 INFO:tasks.workunit.client.0.smithi181.stderr:450812: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-10T21:52:31.436 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-10T21:52:31.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-10T21:52:31.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T21:52:31.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:52:31.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:52:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:31.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-10T21:52:32.253 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-10T21:52:32.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:32.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:52:32.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:32.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:52:32.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:52:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:52:08.116396+0000 '>' 2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:32.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:52:33.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:52:33.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:52:33.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:52:33.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:52:33.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:52:33.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:52:33.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:52:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:52:08.116396+0000 '>' 2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:52:34.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:52:34.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:52:34.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:52:34.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:52:34.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:52:34.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:52:34.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:52:35.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:52:32.277603+0000 '>' 2024-09-10T21:52:08.116396+0000 2024-09-10T21:52:35.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:52:35.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-10T21:52:37.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-10T21:52:37.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-10T21:52:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-10T21:52:37.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-10T21:52:37.474 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:52:37.474 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:52:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-10T21:52:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-10T21:52:37.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-10T21:52:37.811 INFO:tasks.workunit.client.0.smithi181.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 64s) 2024-09-10T21:52:37.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-10T21:52:37.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-10T21:52:38.156 INFO:tasks.workunit.client.0.smithi181.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 65s) 2024-09-10T21:52:38.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-10T21:52:38.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-10T21:52:38.513 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:52:38.513 INFO:tasks.workunit.client.0.smithi181.stdout: 1/1 objects unfound (100.000%) 2024-09-10T21:52:38.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:38.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:38.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:38.624 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:52:38.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:52:38.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:52:38.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:52:38.626 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:52:38.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:52:38.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:52:38.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:52:38.628 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:52:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:52:38.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:52:38.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:52:38.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:38.650 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:52:38.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-repair 0 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-10T21:52:38.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:38.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:38.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:38.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:52:38.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:52:38.656 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:52:38.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:52:38.657 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:52:38.657 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:52:38.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:52:38.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:52:38.659 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:52:38.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T21:52:38.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-10T21:52:38.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:52:38.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:38.662 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52449 2024-09-10T21:52:38.662 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52449 2024-09-10T21:52:38.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:52:38.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:52:38.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T21:52:38.664 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T21:52:38.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T21:52:38.731 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-09-10T21:52:38.731 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-09-10T21:52:38.792 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T21:52:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T21:52:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T21:52:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T21:52:38.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-snaps 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-09-10T21:52:38.799 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-09-10T21:52:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none ' 2024-09-10T21:52:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-09-10T21:52:38.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-10T21:52:38.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-09-10T21:52:38.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_snaps_primary 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_snaps_replica' 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-10T21:52:38.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:52:38.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:52:38.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:52:38.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:52:38.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:52:38.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:52:38.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:52:38.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:52:38.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:52:38.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:52:38.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:52:38.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:52:38.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:52:38.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:52:38.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:52:38.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:38.809 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:38.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:52:38.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:52:38.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:52:38.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-10T21:52:38.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:52:38.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:38.811 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:38.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.455221 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-09-10T21:52:38.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.455221 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-10T21:52:38.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-09-10T21:52:39.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:52:39.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:39.267 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:39.267 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:39.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:39.268 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:39.268 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:52:39.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-09-10T21:52:39.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:52:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:52:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:52:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:52:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get fsid 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:52:39.542 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get mon_host 2024-09-10T21:52:39.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-10T21:52:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:52:40.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:52:40.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:40.437 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:40.437 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:40.438 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:52:40.438 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:52:40.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:52:40.465 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-09-10T21:52:40.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-09-10T21:52:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:52:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-09-10T21:52:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:52:40.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:52:40.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:52:40.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:52:40.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=247c8238-49c3-46bb-918a-bec0fb2c7b02 2024-09-10T21:52:40.474 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 247c8238-49c3-46bb-918a-bec0fb2c7b02 2024-09-10T21:52:40.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 247c8238-49c3-46bb-918a-bec0fb2c7b02' 2024-09-10T21:52:40.474 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:52:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCov+Bmvzn5HBAAZF0WjOG92apx95K3ys3AWA== 2024-09-10T21:52:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCov+Bmvzn5HBAAZF0WjOG92apx95K3ys3AWA=="}' 2024-09-10T21:52:40.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 247c8238-49c3-46bb-918a-bec0fb2c7b02 -i td/osd-scrub-snaps/0/new.json 2024-09-10T21:52:40.692 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:52:40.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-10T21:52:40.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCov+Bmvzn5HBAAZF0WjOG92apx95K3ys3AWA== --osd-uuid 247c8238-49c3-46bb-918a-bec0fb2c7b02 2024-09-10T21:52:40.731 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:40.730+0000 7f316ee0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:40.743 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:40.743+0000 7f316ee0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:40.744 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:40.744+0000 7f316ee0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:40.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:40.744+0000 7f316ee0d640 -1 bdev(0x562c781c3c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:52:40.745 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:40.744+0000 7f316ee0d640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-10T21:52:46.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-10T21:52:46.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:52:46.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:52:46.478 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:52:46.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:52:46.833 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:52:46.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:52:46.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:52:46.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:52:46.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:52:46.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:52:46.859 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:46.857+0000 7feb62fc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:46.861 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:46.859+0000 7feb62fc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:46.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:46.860+0000 7feb62fc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:52:47.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:52:47.105 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:52:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:47.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:47.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:47.748 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:47.746+0000 7feb62fc3640 -1 Falling back to public interface 2024-09-10T21:52:48.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:48.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:48.382 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:52:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:52:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:48.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:48.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:49.323 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:52:49.321+0000 7feb62fc3640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:52:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:49.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:49.667 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:52:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:52:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:49.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:49.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:50.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:50.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:50.953 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:52:50.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:52:50.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:50.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:51.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:52.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:52.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:52.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:52:52.251 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:52:52.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:52.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:52.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:53.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:53.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:53.539 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:52:53.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:52:53.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:53.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:53.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:54.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:54.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:54.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:52:54.893 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:52:54.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:54.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:55.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:56.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:56.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:56.190 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:52:56.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:52:56.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:56.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:56.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:57.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:57.475 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:57.476 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:52:57.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:52:57.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:57.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:57.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:52:58.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:52:58.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:52:58.762 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:52:58.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:52:58.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:52:58.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:52:59.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:00.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:00.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:00.105 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:53:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:53:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:00.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:00.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:01.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:01.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:01.390 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:53:01.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:53:01.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:01.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:01.827 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:02.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:02.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:02.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:53:02.829 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:53:02.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:02.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:03.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:04.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:04.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:04.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:53:04.422 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:53:04.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:04.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:05.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:06.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:06.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:06.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:53:06.357 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:53:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:06.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:06.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:07.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:07.809 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:53:07.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:53:07.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:07.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:08.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:09.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:09.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:09.163 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:53:09.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:53:09.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:09.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:09.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:53:10.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:53:10.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:53:10.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:53:10.848 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:53:10.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:53:10.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:53:11.165 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4110781759,v1:127.0.0.1:6803/4110781759] [v2:127.0.0.1:6804/4110781759,v1:127.0.0.1:6805/4110781759] exists,up 247c8238-49c3-46bb-918a-bec0fb2c7b02 2024-09-10T21:53:11.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:53:11.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:53:11.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:53:11.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-09-10T21:53:11.747 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:53:11.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-09-10T21:53:12.157 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:53:12.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-09-10T21:53:12.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T21:53:12.668 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T21:53:12.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:53:13.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-09-10T21:53:13.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:53:13.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:53:13.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:53:13.682 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:53:13.683 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:53:13.683 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:53:13.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:53:13.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:53:13.683 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:53:13.778 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:53:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:53:14.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:53:14.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:53:14.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836483 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836483 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836483' 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836483 2024-09-10T21:53:14.149 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:53:14.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:53:14.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836483 2024-09-10T21:53:14.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:53:14.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836483 2024-09-10T21:53:14.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-09-10T21:53:14.152 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836483 2024-09-10T21:53:14.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:53:14.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836482 -lt 21474836483 2024-09-10T21:53:14.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:53:15.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:53:15.429 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:53:15.713 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836483 2024-09-10T21:53:15.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:53:15.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:53:15.714 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:53:16.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:53:16.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:53:16.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:53:16.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:53:16.321 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:53:16.321 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:53:16.322 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:53:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:53:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:53:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:53:16.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-09-10T21:53:16.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-09-10T21:53:16.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-09-10T21:53:16.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-09-10T21:53:16.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.455221 bs=1032 count=1 2024-09-10T21:53:16.976 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:53:16.976 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:53:16.976 INFO:tasks.workunit.client.0.smithi181.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5714e-05 s, 22.6 MB/s 2024-09-10T21:53:16.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-09-10T21:53:16.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:16.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.455221 2024-09-10T21:53:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.455221 2024-09-10T21:53:17.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.455221 2024-09-10T21:53:17.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.455221 2024-09-10T21:53:17.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.455221 2024-09-10T21:53:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.455221 2024-09-10T21:53:17.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.455221 2024-09-10T21:53:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.455221 2024-09-10T21:53:17.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.656 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.455221 2024-09-10T21:53:17.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.455221 2024-09-10T21:53:17.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.455221 2024-09-10T21:53:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.455221 2024-09-10T21:53:17.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:17.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.455221 2024-09-10T21:53:18.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:18.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.455221 2024-09-10T21:53:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:18.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.455221 2024-09-10T21:53:18.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-10T21:53:18.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.455221 2024-09-10T21:53:18.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-09-10T21:53:18.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T21:53:18.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:53:18.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T21:53:18.259 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:53:18.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-09-10T21:53:18.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.455221 0 2024-09-10T21:53:18.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-10T21:53:18.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-10T21:53:18.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.455221 2024-09-10T21:53:18.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-10T21:53:18.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-10T21:53:18.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-10T21:53:18.799 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap1 2024-09-10T21:53:18.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=1 2024-09-10T21:53:18.802 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:53:18.802 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:53:18.802 INFO:tasks.workunit.client.0.smithi181.stderr:256 bytes copied, 6.2565e-05 s, 4.1 MB/s 2024-09-10T21:53:18.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.455221 2024-09-10T21:53:18.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:53:19.035 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:53:19.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-10T21:53:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.455221 2024-09-10T21:53:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.455221 2024-09-10T21:53:19.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.455221 2024-09-10T21:53:19.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.455221 2024-09-10T21:53:19.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.455221 2024-09-10T21:53:19.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.455221 2024-09-10T21:53:19.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.455221 2024-09-10T21:53:19.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.455221 2024-09-10T21:53:19.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:53:19.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.455221 2024-09-10T21:53:19.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-10T21:53:19.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-10T21:53:19.997 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap2 2024-09-10T21:53:19.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=2 2024-09-10T21:53:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records in 2024-09-10T21:53:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records out 2024-09-10T21:53:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:512 bytes copied, 6.4327e-05 s, 8.0 MB/s 2024-09-10T21:53:20.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:53:20.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-10T21:53:20.140 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-10T21:53:20.423 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap3 2024-09-10T21:53:20.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=3 2024-09-10T21:53:20.426 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records in 2024-09-10T21:53:20.426 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records out 2024-09-10T21:53:20.426 INFO:tasks.workunit.client.0.smithi181.stderr:768 bytes copied, 6.7469e-05 s, 11.4 MB/s 2024-09-10T21:53:20.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:53:20.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-10T21:53:20.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-10T21:53:20.759 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap4 2024-09-10T21:53:20.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=4 2024-09-10T21:53:20.762 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records in 2024-09-10T21:53:20.762 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records out 2024-09-10T21:53:20.762 INFO:tasks.workunit.client.0.smithi181.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.8925e-05 s, 14.9 MB/s 2024-09-10T21:53:20.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:53:20.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.455221 2024-09-10T21:53:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-10T21:53:21.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-10T21:53:21.161 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap5 2024-09-10T21:53:21.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-10T21:53:21.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-10T21:53:21.360 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap6 2024-09-10T21:53:21.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=6 2024-09-10T21:53:21.363 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records in 2024-09-10T21:53:21.363 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records out 2024-09-10T21:53:21.364 INFO:tasks.workunit.client.0.smithi181.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.0874e-05 s, 21.7 MB/s 2024-09-10T21:53:21.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:53:21.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-10T21:53:21.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-10T21:53:21.645 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap7 2024-09-10T21:53:21.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-10T21:53:21.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-10T21:53:21.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:53:21.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:53:22.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:53:22.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-10T21:53:22.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:22.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-10T21:53:24.312 INFO:tasks.workunit.client.0.smithi181.stdout:WARNING: only removing head with clones present 2024-09-10T21:53:24.312 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T21:53:24.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:53:24.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-10T21:53:25.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:25.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:53:26.573 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:2# 2024-09-10T21:53:27.088 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:53:27.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-10T21:53:27.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:27.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:27.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:53:28.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:53:28.736 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.737 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:28.738 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:28.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:53:28.740 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:28.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:53:29.977 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:1# 2024-09-10T21:53:30.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:53:31.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:53:31.545 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.546 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:31.547 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:31.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:53:31.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:31.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:53:31.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:53:31.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-10T21:53:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=18 2024-09-10T21:53:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records in 2024-09-10T21:53:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records out 2024-09-10T21:53:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000128145 s, 36.0 MB/s 2024-09-10T21:53:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:53:33.627 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-10T21:53:34.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:34.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=15 2024-09-10T21:53:34.209 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records in 2024-09-10T21:53:34.210 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records out 2024-09-10T21:53:34.210 INFO:tasks.workunit.client.0.smithi181.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000149633 s, 25.7 MB/s 2024-09-10T21:53:34.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:53:35.964 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-10T21:53:35.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-10T21:53:36.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:36.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:53:37.493 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-10T21:53:38.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:53:39.318 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:53:39.320 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:39.321 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:39.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-10T21:53:39.323 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-10T21:53:39.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:39.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:53:40.964 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-10T21:53:41.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:53:42.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:53:42.530 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:53:42.530 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:53:42.530 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:53:42.531 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:53:42.532 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:53:42.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:53:42.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-10T21:53:43.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:43.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-10T21:53:44.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:44.582 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-10T21:53:44.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:44.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=7 2024-09-10T21:53:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records in 2024-09-10T21:53:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records out 2024-09-10T21:53:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.2623e-05 s, 19.3 MB/s 2024-09-10T21:53:44.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:53:46.091 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-10T21:53:46.408 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:46.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:46.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-10T21:53:48.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-10T21:53:48.467 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:48.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:48.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-10T21:53:50.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-10T21:53:50.993 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:51.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:51.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-10T21:53:52.691 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-10T21:53:53.005 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:53.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:53.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-10T21:53:54.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-10T21:53:55.025 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:55.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:55.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-10T21:53:57.015 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-10T21:53:57.352 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:57.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-10T21:53:59.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-10T21:53:59.399 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:53:59.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:53:59.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-10T21:54:01.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-10T21:54:01.566 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:54:01.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:54:01.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-10T21:54:03.387 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-10T21:54:03.707 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:54:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:54:03.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-10T21:54:05.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-10T21:54:05.721 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-10T21:54:06.052 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:54:06.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:54:06.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-10T21:54:07.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-10T21:54:07.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-10T21:54:07.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.455221 2024-09-10T21:54:07.845 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-09-10T21:54:07.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-09-10T21:54:07.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:54:07.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:54:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:54:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:54:07.853 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:54:07.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:54:07.878 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:07.878+0000 7f9ebf41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:07.881 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:07.880+0000 7f9ebf41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:07.882 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:07.881+0000 7f9ebf41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:54:08.134 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:54:08.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:08.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:08.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:08.767 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:08.766+0000 7f9ebf41c640 -1 Falling back to public interface 2024-09-10T21:54:09.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:09.415 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:54:09.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:09.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:54:09.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:09.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:09.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:10.211 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:10.210+0000 7f9ebf41c640 -1 osd.0 18 log_to_monitors true 2024-09-10T21:54:10.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:10.717 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:54:10.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:10.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:54:10.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:10.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:10.994 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/2163665219,v1:127.0.0.1:6803/2163665219] [v2:127.0.0.1:6804/2163665219,v1:127.0.0.1:6805/2163665219] exists,up 247c8238-49c3-46bb-918a-bec0fb2c7b02 2024-09-10T21:54:10.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:54:10.994 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:54:10.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:54:10.995 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-09-10T21:54:11.088 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:11.089 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T21:54:11.089 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:11.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-09-10T21:54:11.233 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:11.233 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:54:11.234 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:11.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-10T21:54:11.336 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:11.336 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-10T21:54:11.336 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:11.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-10T21:54:11.439 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:11.440 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-10T21:54:11.440 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:11.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:54:11.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:54:11.535 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:54:11.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T21:54:11.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:54:11.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:54:11.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313218 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313218 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-90194313218' 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-90194313218 2024-09-10T21:54:11.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:54:11.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:54:11.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-90194313218 2024-09-10T21:54:11.905 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:54:11.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313218 2024-09-10T21:54:11.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-09-10T21:54:11.906 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 90194313218 2024-09-10T21:54:11.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:54:12.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 90194313218 2024-09-10T21:54:12.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:54:13.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:54:13.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:54:13.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313218 -lt 90194313218 2024-09-10T21:54:13.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:54:13.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:54:13.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:54:13.800 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:54:14.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:54:14.070 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:54:14.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:54:14.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:54:14.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:54:14.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:54:14.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:54:14.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-09-10T21:54:14.503 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:14.503 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-09-10T21:54:14.503 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:14.511 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-09-10T21:54:14.606 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:14.606 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-09-10T21:54:14.606 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:14.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2024-09-10T21:54:14.714 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:14.715 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2024-09-10T21:54:14.715 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:14.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2024-09-10T21:54:14.814 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:14.814 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2024-09-10T21:54:14.815 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:14.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-09-10T21:54:14.916 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:14.916 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_chunk_max": "25" 2024-09-10T21:54:14.916 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:14.925 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-09-10T21:54:15.066 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:54:15.066 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_chunk_min": "5" 2024-09-10T21:54:15.066 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:15.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:15.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:15.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T21:54:15.622 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-10T21:54:15.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:15.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:54:15.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:15.635 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:15.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:53:12.550909+0000 '>' 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:15.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:54:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:54:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:54:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:54:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:16.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:17.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:53:12.550909+0000 '>' 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:17.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:54:18.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:54:18.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:54:18.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:54:18.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:18.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:18.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:18.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:18.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:53:12.550909+0000 '>' 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:18.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:54:19.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:54:19.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:54:19.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:54:19.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:19.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:19.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:19.459 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:19.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:53:12.550909+0000 '>' 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:19.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:54:20.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:54:20.741 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:54:21.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:54:16.225394+0000 '>' 2024-09-10T21:53:12.550909+0000 2024-09-10T21:54:21.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:54:21.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:21.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2024-09-10T21:54:21.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2024-09-10T21:54:21.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-09-10T21:54:21.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-09-10T21:54:21.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2024-09-10T21:54:21.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-10T21:54:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2024-09-10T21:54:21.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-09-10T21:54:21.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2024-09-10T21:54:21.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:54:21.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-10T21:54:21.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:54:21.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:54:21.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:54:21.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-09-10T21:54:21.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2024-09-10T21:54:21.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:54:21.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-10T21:54:21.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:54:21.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:54:21.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:54:21.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2024-09-10T21:54:21.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2024-09-10T21:54:21.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-09-10T21:54:21.197 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-09-10T21:54:21.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-09-10T21:54:21.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2024-09-10T21:54:21.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-09-10T21:54:21.198 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-09-10T21:54:21.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='470853 ' 2024-09-10T21:54:21.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2024-09-10T21:54:21.199 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2024-09-10T21:54:21.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:21.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-09-10T21:54:21.355 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap1 2024-09-10T21:54:21.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:21.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-09-10T21:54:21.571 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap2 2024-09-10T21:54:21.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:21.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-09-10T21:54:21.772 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap3 2024-09-10T21:54:21.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:21.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-09-10T21:54:21.973 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap4 2024-09-10T21:54:21.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:21.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-09-10T21:54:22.236 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap5 2024-09-10T21:54:22.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:22.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-09-10T21:54:22.425 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap6 2024-09-10T21:54:22.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-10T21:54:22.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-09-10T21:54:22.626 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap7 2024-09-10T21:54:22.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2024-09-10T21:54:27.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2024-09-10T21:54:27.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-09-10T21:54:27.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2024-09-10T21:54:27.895 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:54:27.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2024-09-10T21:54:27.907 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2024-09-10T21:54:28.161 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:54:28.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2024-09-10T21:54:28.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2024-09-10T21:54:28.428 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T21:54:28.429 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-09-10T21:54:22.651794+0000 23'60 29:139 [0] 0 [0] 0 18'56 2024-09-10T21:54:16.225394+0000 0'0 2024-09-10T21:53:12.550909+0000 7 1 periodic scrub scheduled @ 2024-09-12T06:21:06.524352+0000 33 0 2024-09-10T21:54:28.429 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T21:54:28.429 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T21:54:28.429 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:54:28.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2024-09-10T21:54:28.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 470853 2024-09-10T21:54:28.440 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-09-10T21:54:28.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:54:28.441 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:54:28.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:54:28.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:54:28.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:54:33.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:54:33.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-09-10T21:54:33.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-10T21:54:33.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.772 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.793 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.795 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.802 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-10T21:54:33.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:54:33.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-09-10T21:54:33.807 INFO:tasks.workunit.client.0.smithi181.stdout:TEST PASSED 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:54:33.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:54:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:54:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:54:33.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:54:33.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:54:33.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:54:33.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:54:33.813 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:54:33.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:54:33.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:54:33.815 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:54:33.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:54:33.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:54:33.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:54:33.816 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:54:33.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:54:33.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:54:33.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:54:33.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:54:33.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:33.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:33.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:54:33.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:54:33.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:54:33.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:54:33.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:54:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:54:33.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:54:33.834 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:54:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:54:33.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:54:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:54:33.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:54:33.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:54:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:54:33.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:54:33.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:54:33.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:33.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:33.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:54:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:54:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:54:33.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-10T21:54:33.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:54:33.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:33.841 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:33.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.455221 2024-09-10T21:54:33.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:54:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:54:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:54:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-10T21:54:33.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.455221 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:54:33.845 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:54:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-10T21:54:33.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-10T21:54:34.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:54:34.319 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:34.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:54:34.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:54:34.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:54:34.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:54:34.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:54:34.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:54:34.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:54:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:54:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:54:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:54:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:54:34.522 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:34.523 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:34.523 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:54:34.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:54:34.523 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get fsid 2024-09-10T21:54:34.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:54:34.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:54:34.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:34.605 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:54:34.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:54:34.606 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get mon_host 2024-09-10T21:54:34.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-10T21:54:34.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-10T21:54:34.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:54:34.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:54:34.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:54:34.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-10T21:54:34.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:35.513 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:35.514 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:54:35.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:54:35.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-10T21:54:35.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:54:35.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:54:35.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:54:35.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:54:35.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:54:35.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db800462-48cc-4185-8184-eed41997a0da 2024-09-10T21:54:35.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 db800462-48cc-4185-8184-eed41997a0da' 2024-09-10T21:54:35.558 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 db800462-48cc-4185-8184-eed41997a0da 2024-09-10T21:54:35.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:54:35.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbwOBmJHubIhAAjjepXEjGYnO9lLsVCkvvYA== 2024-09-10T21:54:35.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbwOBmJHubIhAAjjepXEjGYnO9lLsVCkvvYA=="}' 2024-09-10T21:54:35.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db800462-48cc-4185-8184-eed41997a0da -i td/osd-scrub-snaps/0/new.json 2024-09-10T21:54:35.911 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:54:35.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-10T21:54:35.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbwOBmJHubIhAAjjepXEjGYnO9lLsVCkvvYA== --osd-uuid db800462-48cc-4185-8184-eed41997a0da 2024-09-10T21:54:35.962 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:35.961+0000 7fcb958d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:35.964 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:35.964+0000 7fcb958d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:35.965 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:35.965+0000 7fcb958d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:35.965+0000 7fcb958d6640 -1 bdev(0x557319ccdc00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:54:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:35.965+0000 7fcb958d6640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-10T21:54:41.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-10T21:54:41.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:54:41.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:54:41.466 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:54:41.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:54:41.837 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:54:41.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:54:41.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:54:41.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:54:41.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:54:41.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:54:41.876 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:41.875+0000 7fea4aa31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:41.915 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:41.914+0000 7fea4aa31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:41.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:41.915+0000 7fea4aa31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:54:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:54:42.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:54:42.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:54:42.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:42.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:42.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:43.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:43.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:43.454 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:54:43.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:54:43.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:43.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:43.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:43.765 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:43.765+0000 7fea4aa31640 -1 Falling back to public interface 2024-09-10T21:54:44.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:44.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:44.747 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:54:44.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:54:44.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:44.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:45.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:45.624 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:54:45.623+0000 7fea4aa31640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:54:46.042 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:46.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:46.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:54:46.043 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:54:46.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:46.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:46.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:47.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:47.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:47.524 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:54:47.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:54:47.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:47.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:47.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:48.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:48.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:48.818 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:54:48.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:54:48.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:48.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:49.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:50.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:50.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:50.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:54:50.154 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:54:50.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:50.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:50.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:51.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:51.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:51.450 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:54:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:54:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:51.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:51.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:52.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:52.748 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:52.748 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:54:52.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:54:52.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:52.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:53.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:54.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:54.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:54.047 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:54:54.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:54:54.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:54.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:54.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:55.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:55.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:55.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:54:55.403 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:54:55.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:55.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:55.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:56.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:56.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:56.703 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:54:56.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:54:56.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:56.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:57.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:58.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:58.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:58.004 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:54:58.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:54:58.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:58.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:54:58.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:54:59.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:54:59.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:54:59.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:54:59.329 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:54:59.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:54:59.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:00.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:01.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:01.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:01.136 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:55:01.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:55:01.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:01.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:01.679 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:02.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:02.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:02.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:55:02.681 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:55:02.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:02.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:03.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:04.400 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:55:04.400 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:55:04.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:04.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:05.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:06.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:06.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:06.021 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:55:06.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:55:06.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:06.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:06.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:07.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:07.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:07.762 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:55:07.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:55:07.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:07.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:55:08.121 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3897150024,v1:127.0.0.1:6803/3897150024] [v2:127.0.0.1:6804/3897150024,v1:127.0.0.1:6805/3897150024] exists,up db800462-48cc-4185-8184-eed41997a0da 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:55:08.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:55:08.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:55:08.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:55:08.128 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 ebb5371f-3471-4a9e-8d09-0561708910dc 2024-09-10T21:55:08.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ebb5371f-3471-4a9e-8d09-0561708910dc 2024-09-10T21:55:08.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ebb5371f-3471-4a9e-8d09-0561708910dc' 2024-09-10T21:55:08.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:55:08.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8wOBm+X36CBAAxHnE3ef63ZU4FzBrKdlEjg== 2024-09-10T21:55:08.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8wOBm+X36CBAAxHnE3ef63ZU4FzBrKdlEjg=="}' 2024-09-10T21:55:08.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ebb5371f-3471-4a9e-8d09-0561708910dc -i td/osd-scrub-snaps/1/new.json 2024-09-10T21:55:08.494 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:55:08.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-10T21:55:08.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8wOBm+X36CBAAxHnE3ef63ZU4FzBrKdlEjg== --osd-uuid ebb5371f-3471-4a9e-8d09-0561708910dc 2024-09-10T21:55:08.547 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:08.547+0000 7f48fefa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:08.578 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:08.577+0000 7f48fefa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:08.579 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:08.578+0000 7f48fefa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:08.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:08.579+0000 7f48fefa8640 -1 bdev(0x55d3ec4e9c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:55:08.580 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:08.579+0000 7f48fefa8640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-10T21:55:14.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-10T21:55:14.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:55:14.851 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:55:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:55:14.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:55:15.226 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:55:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:55:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:55:15.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:55:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:55:15.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:55:15.265 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:15.263+0000 7f7d6cb51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:15.292 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:15.290+0000 7f7d6cb51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:15.293 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:15.291+0000 7f7d6cb51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:55:15.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:15.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:55:15.572 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:55:15.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:15.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:15.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:16.398 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:16.396+0000 7f7d6cb51640 -1 Falling back to public interface 2024-09-10T21:55:16.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:16.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:16.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:55:16.902 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:55:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:16.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:17.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:17.935 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:55:17.933+0000 7f7d6cb51640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:55:18.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:18.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:18.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:55:18.207 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:55:18.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:18.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:18.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:19.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:19.672 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:55:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:55:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:19.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:20.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:21.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:21.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:21.041 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:55:21.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:55:21.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:21.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:21.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:22.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:22.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:22.338 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:55:22.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:55:22.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:22.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:22.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:23.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:23.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:23.639 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:55:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:55:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:23.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:24.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:25.043 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:55:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:55:25.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:25.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:25.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:26.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:26.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:26.343 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:55:26.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:55:26.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:26.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:26.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:27.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:27.644 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:27.644 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:55:27.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:55:27.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:27.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:28.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:29.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:29.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:29.016 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:55:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:55:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:29.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:29.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:30.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:30.316 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:55:30.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:30.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:55:30.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:30.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:30.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:31.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:31.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:31.744 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:55:31.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:55:31.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:31.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:32.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:33.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:33.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:33.448 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:55:33.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:55:33.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:33.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:33.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:34.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:34.823 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:55:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:55:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:34.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:35.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:36.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:36.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:36.389 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:55:36.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:55:36.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:36.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:37.037 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:38.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:38.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:38.039 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:55:38.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:55:38.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:38.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:38.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:39.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:39.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:39.600 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:55:39.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:55:39.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:39.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:40.266 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:55:41.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:55:41.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:55:41.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:55:41.268 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:55:41.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:55:41.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:55:41.580 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4109812783,v1:127.0.0.1:6811/4109812783] [v2:127.0.0.1:6812/4109812783,v1:127.0.0.1:6813/4109812783] exists,up ebb5371f-3471-4a9e-8d09-0561708910dc 2024-09-10T21:55:41.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:55:41.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:55:41.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:55:41.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-10T21:55:41.994 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:55:42.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-10T21:55:42.521 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:55:42.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-10T21:55:42.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T21:55:42.940 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T21:55:42.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:55:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-10T21:55:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:55:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:55:43.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:55:43.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:55:43.955 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:55:43.955 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:55:43.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:55:43.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:55:43.955 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:55:44.053 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:55:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:55:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:55:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:55:44.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:55:44.343 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:55:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-10T21:55:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-10T21:55:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-10T21:55:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:55:44.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:55:44.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672968 2024-09-10T21:55:44.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672968 2024-09-10T21:55:44.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2024-09-10T21:55:44.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:55:44.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-10T21:55:44.575 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:55:44.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:55:44.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-10T21:55:44.576 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:55:44.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-10T21:55:44.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-10T21:55:44.577 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836490 2024-09-10T21:55:44.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:55:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-10T21:55:44.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:55:45.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:55:45.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:55:46.159 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-10T21:55:46.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:55:46.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672968 2024-09-10T21:55:46.160 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:55:46.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:55:46.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672968 2024-09-10T21:55:46.161 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:55:46.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672968 2024-09-10T21:55:46.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-10T21:55:46.163 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672968 2024-09-10T21:55:46.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:55:46.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-10T21:55:46.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:55:46.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:55:46.490 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:55:46.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:55:46.885 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:55:47.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:55:47.200 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:55:47.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:55:47.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:55:47.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:55:47.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:55:47.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:55:47.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-10T21:55:47.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-10T21:55:47.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-10T21:55:47.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-10T21:55:47.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.455221 bs=1032 count=1 2024-09-10T21:55:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:55:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:55:47.904 INFO:tasks.workunit.client.0.smithi181.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5621e-05 s, 22.6 MB/s 2024-09-10T21:55:47.904 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-10T21:55:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:47.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.455221 2024-09-10T21:55:48.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.455221 2024-09-10T21:55:48.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.455221 2024-09-10T21:55:48.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.455221 2024-09-10T21:55:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.455221 2024-09-10T21:55:48.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.455221 2024-09-10T21:55:48.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.455221 2024-09-10T21:55:48.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.455221 2024-09-10T21:55:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:48.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.455221 2024-09-10T21:55:49.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.084 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.455221 2024-09-10T21:55:49.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.455221 2024-09-10T21:55:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.455221 2024-09-10T21:55:49.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.455221 2024-09-10T21:55:49.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.455221 2024-09-10T21:55:49.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.455221 2024-09-10T21:55:49.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:55:49.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.455221 2024-09-10T21:55:50.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-10T21:55:50.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T21:55:50.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:55:50.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T21:55:50.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T21:55:50.407 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:55:50.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:55:50.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-10T21:55:50.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:55:51.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-10T21:55:51.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.455221 '$primary' 2024-09-10T21:55:51.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.455221 1 2024-09-10T21:55:51.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-10T21:55:51.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-10T21:55:51.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.455221 2024-09-10T21:55:51.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-09-10T21:55:51.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-10T21:55:51.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-10T21:55:51.282 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap1 2024-09-10T21:55:51.284 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=1 2024-09-10T21:55:51.285 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:55:51.285 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:55:51.285 INFO:tasks.workunit.client.0.smithi181.stderr:256 bytes copied, 6.6795e-05 s, 3.8 MB/s 2024-09-10T21:55:51.285 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.455221 2024-09-10T21:55:51.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:55:51.685 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:55:51.836 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-10T21:55:51.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:51.837 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.455221 2024-09-10T21:55:51.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:51.970 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.455221 2024-09-10T21:55:52.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.455221 2024-09-10T21:55:52.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.229 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.455221 2024-09-10T21:55:52.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.455221 2024-09-10T21:55:52.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.455221 2024-09-10T21:55:52.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.455221 2024-09-10T21:55:52.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.740 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.455221 2024-09-10T21:55:52.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:55:52.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.455221 2024-09-10T21:55:53.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-10T21:55:53.042 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-10T21:55:53.208 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap2 2024-09-10T21:55:53.209 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=2 2024-09-10T21:55:53.210 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records in 2024-09-10T21:55:53.210 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records out 2024-09-10T21:55:53.211 INFO:tasks.workunit.client.0.smithi181.stderr:512 bytes copied, 6.8508e-05 s, 7.5 MB/s 2024-09-10T21:55:53.211 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:55:53.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-10T21:55:53.443 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-10T21:55:53.659 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap3 2024-09-10T21:55:53.660 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=3 2024-09-10T21:55:53.665 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records in 2024-09-10T21:55:53.665 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records out 2024-09-10T21:55:53.665 INFO:tasks.workunit.client.0.smithi181.stderr:768 bytes copied, 6.9985e-05 s, 11.0 MB/s 2024-09-10T21:55:53.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:55:53.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-10T21:55:53.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-10T21:55:54.137 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap4 2024-09-10T21:55:54.139 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=4 2024-09-10T21:55:54.140 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records in 2024-09-10T21:55:54.140 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records out 2024-09-10T21:55:54.140 INFO:tasks.workunit.client.0.smithi181.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.6625e-05 s, 15.4 MB/s 2024-09-10T21:55:54.140 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:55:54.373 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.455221 2024-09-10T21:55:54.507 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-10T21:55:54.507 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-10T21:55:54.690 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap5 2024-09-10T21:55:54.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-10T21:55:54.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-10T21:55:54.932 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap6 2024-09-10T21:55:54.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=6 2024-09-10T21:55:54.935 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records in 2024-09-10T21:55:54.935 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records out 2024-09-10T21:55:54.935 INFO:tasks.workunit.client.0.smithi181.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.0904e-05 s, 21.7 MB/s 2024-09-10T21:55:54.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:55:55.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-10T21:55:55.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-10T21:55:55.335 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap7 2024-09-10T21:55:55.337 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-10T21:55:55.562 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-10T21:55:55.713 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-10T21:55:55.863 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-10T21:55:55.864 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:55:55.864 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:55:55.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:55:55.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:55:55.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:55:56.170 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:55:56.171 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-09-10T21:55:56.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:55:56.812 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-10T21:55:58.343 INFO:tasks.workunit.client.0.smithi181.stdout:WARNING: only removing head with clones present 2024-09-10T21:55:58.343 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T21:55:58.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-10T21:55:58.858 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-10T21:55:59.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:55:59.450 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:56:00.755 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:2# 2024-09-10T21:56:01.270 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-10T21:56:01.271 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-10T21:56:01.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:01.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:01.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-10T21:56:02.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:56:02.940 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:02.941 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.942 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:56:02.943 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:02.944 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:56:04.058 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:1# 2024-09-10T21:56:04.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-10T21:56:05.637 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:56:05.638 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.638 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:56:05.639 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:05.640 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:05.641 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:05.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:56:05.642 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:05.642 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:56:05.642 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-10T21:56:05.642 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-10T21:56:06.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:06.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=18 2024-09-10T21:56:06.278 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records in 2024-09-10T21:56:06.278 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records out 2024-09-10T21:56:06.278 INFO:tasks.workunit.client.0.smithi181.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000135888 s, 33.9 MB/s 2024-09-10T21:56:06.278 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:56:07.754 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-09-10T21:56:08.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:08.358 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=15 2024-09-10T21:56:08.359 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records in 2024-09-10T21:56:08.359 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records out 2024-09-10T21:56:08.359 INFO:tasks.workunit.client.0.smithi181.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000139479 s, 27.5 MB/s 2024-09-10T21:56:08.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:56:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-09-10T21:56:09.878 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-10T21:56:10.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:10.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:56:11.459 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-10T21:56:12.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-10T21:56:13.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:56:13.289 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:13.290 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:13.291 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:56:13.292 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-09-10T21:56:13.293 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-10T21:56:13.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:13.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:56:14.820 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-10T21:56:15.335 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-10T21:56:16.396 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:56:16.397 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:16.398 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:16.399 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:56:16.399 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:16.399 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:56:16.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:56:16.399 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:56:16.400 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-09-10T21:56:17.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:17.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-10T21:56:18.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:18.513 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-10T21:56:18.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:18.515 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=7 2024-09-10T21:56:18.516 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records in 2024-09-10T21:56:18.516 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records out 2024-09-10T21:56:18.516 INFO:tasks.workunit.client.0.smithi181.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.6711e-05 s, 18.5 MB/s 2024-09-10T21:56:18.516 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:56:20.001 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-09-10T21:56:20.333 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:20.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:20.601 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-10T21:56:22.099 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-09-10T21:56:22.425 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:22.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:22.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-10T21:56:24.440 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-09-10T21:56:24.766 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:25.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:25.043 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-10T21:56:26.724 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-09-10T21:56:27.058 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:27.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:27.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-10T21:56:28.962 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-09-10T21:56:29.291 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:29.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:29.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-10T21:56:31.336 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-09-10T21:56:31.714 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:31.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:31.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-10T21:56:33.389 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-09-10T21:56:33.736 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:34.004 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:34.005 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-10T21:56:35.433 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-09-10T21:56:35.765 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:36.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:36.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-10T21:56:37.427 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-09-10T21:56:37.758 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:38.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:38.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-10T21:56:39.712 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-10T21:56:39.712 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-09-10T21:56:40.041 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:56:40.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:56:40.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-10T21:56:41.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-10T21:56:41.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-10T21:56:41.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.455221 2024-09-10T21:56:41.968 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-10T21:56:41.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:56:41.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:56:41.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:56:41.975 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:56:41.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:56:41.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:56:42.015 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:42.014+0000 7fda4a534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:42.017 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:42.017+0000 7fda4a534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:42.019 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:42.018+0000 7fda4a534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:56:42.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:42.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:56:42.268 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:56:42.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:42.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:56:42.597 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:56:43.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:56:43.599 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:56:43.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:43.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:56:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:43.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:56:43.611 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:43.610+0000 7fda4a534640 -1 Falling back to public interface 2024-09-10T21:56:43.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:56:44.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:56:44.914 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:56:44.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:44.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:56:44.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:44.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:56:45.208 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:56:45.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:45.648+0000 7fda4a534640 -1 osd.0 23 log_to_monitors true 2024-09-10T21:56:46.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:56:46.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:46.210 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:56:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:56:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:46.210 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:56:46.543 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/1641149963,v1:127.0.0.1:6803/1641149963] [v2:127.0.0.1:6804/1641149963,v1:127.0.0.1:6805/1641149963] exists,up db800462-48cc-4185-8184-eed41997a0da 2024-09-10T21:56:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:56:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:56:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:56:46.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-10T21:56:46.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:56:46.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-10T21:56:46.550 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:56:46.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:56:46.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:56:46.551 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-10T21:56:46.551 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:56:46.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:56:46.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:56:46.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:56:46.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:46.584+0000 7fc646818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:46.589 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:46.588+0000 7fc646818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:46.590 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:46.589+0000 7fc646818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:56:46.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:46.915 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:56:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:56:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:46.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:56:47.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:56:47.442 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:47.441+0000 7fc646818640 -1 Falling back to public interface 2024-09-10T21:56:48.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:56:48.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:48.203 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:56:48.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:56:48.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:48.203 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:56:48.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:56:48.963 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:56:48.962+0000 7fc646818640 -1 osd.1 23 log_to_monitors true 2024-09-10T21:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:56:49.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:56:49.504 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:56:49.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:56:49.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:56:49.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:56:49.875 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/2709495991,v1:127.0.0.1:6811/2709495991] [v2:127.0.0.1:6812/2709495991,v1:127.0.0.1:6813/2709495991] exists,up ebb5371f-3471-4a9e-8d09-0561708910dc 2024-09-10T21:56:49.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:56:49.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:56:49.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:56:49.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-10T21:56:49.983 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:56:49.984 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T21:56:49.984 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:49.997 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:56:49.997 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T21:56:49.997 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-10T21:56:50.134 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:56:50.134 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:56:50.134 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.148 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:56:50.148 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:56:50.148 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-10T21:56:50.283 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:56:50.283 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:56:50.284 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.297 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:56:50.297 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:56:50.297 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-10T21:56:50.434 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:56:50.434 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-10T21:56:50.434 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.447 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:56:50.447 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-10T21:56:50.447 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-10T21:56:50.585 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:56:50.585 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-10T21:56:50.585 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.600 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:56:50.601 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-10T21:56:50.601 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:56:50.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-10T21:56:50.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:56:50.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:56:50.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:56:50.607 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:56:50.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:56:50.608 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:56:50.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:56:50.608 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:56:50.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:56:50.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:56:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:56:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:56:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:56:50.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:56:50.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:56:50.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:56:50.983 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:56:50.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:56:50.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:56:50.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:56:51.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-10T21:56:51.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-10T21:56:51.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699' 2024-09-10T21:56:51.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:56:51.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149699 2024-09-10T21:56:51.242 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:56:51.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:56:51.243 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149699 2024-09-10T21:56:51.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:56:51.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-10T21:56:51.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-10T21:56:51.245 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 111669149699 2024-09-10T21:56:51.245 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:56:51.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149699 2024-09-10T21:56:51.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:56:52.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:56:52.532 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:56:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-10T21:56:52.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:56:52.825 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-10T21:56:52.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:56:52.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:56:52.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-10T21:56:52.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:56:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-10T21:56:52.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-10T21:56:52.828 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 124554051586 2024-09-10T21:56:52.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:56:53.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-10T21:56:53.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:56:53.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:56:53.117 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:56:53.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:56:53.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:56:53.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:56:53.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:56:53.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:54.185 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:54.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:54.186 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:56:54.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:54.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T21:56:54.779 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-10T21:56:54.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:54.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:56:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:55:42.887008+0000 '>' 2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:55.079 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:56:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:56:56.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:56:56.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:56:56.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:56.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:56.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:56.081 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:56:56.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:55:42.887008+0000 '>' 2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:56.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:56:57.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:56:57.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:56:57.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:56:57.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:57.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:57.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:57.370 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:56:57.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:55:42.887008+0000 '>' 2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:57.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:56:58.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:56:58.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:56:58.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:56:58.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:58.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:58.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:58.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:56:58.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:55:42.887008+0000 '>' 2024-09-10T21:55:42.887008+0000 2024-09-10T21:56:58.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:56:59.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:56:59.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:56:59.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:56:59.961 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:56:59.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:56:59.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:56:59.962 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:57:00.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:56:55.433612+0000 '>' 2024-09-10T21:55:42.887008+0000 2024-09-10T21:57:00.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:57:00.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:00.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-10T21:57:00.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2024-09-10T21:57:00.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:57:00.258 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-10T21:57:00.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2024-09-10T21:57:00.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-10T21:57:00.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-10T21:57:00.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-10T21:57:00.299 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-10T21:57:00.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-10T21:57:00.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-10T21:57:00.336 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:57:00.336 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch": 29, 2024-09-10T21:57:00.336 INFO:tasks.workunit.client.0.smithi181.stdout: "inconsistents": [ 2024-09-10T21:57:00.336 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.336 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 5 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.337 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 1, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'19", 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'5", 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4241.0:1", 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 5, 2024-09-10T21:57:00.338 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:48.483189+0000", 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:48.483809+0000", 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xaf1a2c9d", 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.339 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.340 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.341 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 2, 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 20 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:57:00.342 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 2, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "18'41", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'20", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4289.0:1", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 20, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:51.586445+0000", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:51.587011+0000", 2024-09-10T21:57:00.343 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.344 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.345 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 4, 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 42 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.346 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch" 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 4, 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.347 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "20'45", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "18'42", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4325.0:1", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 42, 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512, 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:53.246540+0000", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:53.265267+0000", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.348 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x345326c3", 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.349 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.350 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 4608, 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "object_info": { 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 4, 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.351 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "20'45", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "18'42", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4325.0:1", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 42, 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512, 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:53.246540+0000", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:53.265267+0000", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.352 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x345326c3", 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.353 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj4", 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 4 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.354 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj4", 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 7, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2826278768, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "23'51", 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'4", 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4238.0:1", 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 4, 2024-09-10T21:57:00.355 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:48.357656+0000", 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:48.358216+0000", 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xaf1a2c9d", 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.356 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.357 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:57:00.358 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 0 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "missing", 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "info_missing" 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.359 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "info_missing" 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1792 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.360 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj1", 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 18 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.361 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj1", 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1828249343, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'18", 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'1", 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4286.0:1", 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 18, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.362 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:51.320739+0000", 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:51.423301+0000", 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.363 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.364 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj10", 2024-09-10T21:57:00.365 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 32 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.366 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj10", 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 718195851, 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'32", 2024-09-10T21:57:00.367 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'10", 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4307.0:1", 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 32, 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.398799+0000", 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.399418+0000", 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.368 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.369 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.370 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.371 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.372 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.373 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "????", 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.374 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj11", 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.375 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 34 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj11", 2024-09-10T21:57:00.376 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 693400951, 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'34", 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'11", 2024-09-10T21:57:00.377 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4310.0:1", 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 34, 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.524146+0000", 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.524716+0000", 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.378 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.379 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.380 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.381 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.381 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.381 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.382 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.382 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.382 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.383 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.384 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.385 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.386 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj13", 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 38 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.387 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj13", 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.388 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2087409765, 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'38", 2024-09-10T21:57:00.389 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'13", 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4316.0:1", 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 38, 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.775643+0000", 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.776207+0000", 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.390 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.391 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.392 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.393 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.394 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]" 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.395 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.396 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.396 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.396 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.396 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj14", 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.397 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 40 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.398 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj14", 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2484217095, 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'40", 2024-09-10T21:57:00.399 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'14", 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4319.0:1", 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 40, 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.926569+0000", 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.927122+0000", 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.400 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.401 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.402 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.403 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.404 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1033, 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.405 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.406 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj15", 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 15 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_corrupted" 2024-09-10T21:57:00.407 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj15", 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 612772309, 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.408 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "16'15", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "0'0", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4271.0:1", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 15, 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:49.881245+0000", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:49.881855+0000", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.409 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xaf1a2c9d", 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.410 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:57:00.411 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_corrupted" 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.412 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj2", 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.413 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 56 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_missing" 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj2", 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.414 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1058988552, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "23'56", 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "20'48", 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4361.0:1", 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 56, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0, 2024-09-10T21:57:00.415 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:55.775033+0000", 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:55.775600+0000", 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "whiteout", 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty" 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.416 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xffffffff", 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.417 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0, 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 7, 2024-09-10T21:57:00.418 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 4, 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: 4, 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: 3, 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: 2, 2024-09-10T21:57:00.419 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1024, 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.420 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: 7, 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: 6, 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: 5 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.421 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_missing" 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj3", 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.422 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 44 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch" 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.423 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj3", 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1643547569, 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.424 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19'44", 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'22", 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4331.0:1", 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 44, 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768, 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:53.697230+0000", 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:53.697800+0000", 2024-09-10T21:57:00.425 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x94f1f7aa", 2024-09-10T21:57:00.426 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.427 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.428 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.429 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 3840, 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "object_info": { 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj3", 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1643547569, 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.430 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19'44", 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'22", 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4331.0:1", 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 44, 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768, 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:53.697230+0000", 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:53.697800+0000", 2024-09-10T21:57:00.431 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x94f1f7aa", 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.432 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.433 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj6", 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 24 2024-09-10T21:57:00.434 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj6", 2024-09-10T21:57:00.435 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2202164420, 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'24", 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'6", 2024-09-10T21:57:00.436 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4295.0:1", 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 24, 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:51.872642+0000", 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:51.873250+0000", 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.437 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.438 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.439 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.440 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.441 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.442 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj7", 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.443 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 26 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.444 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj7", 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1552453721, 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.445 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'26", 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'7", 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4298.0:1", 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 26, 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.005118+0000", 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.005663+0000", 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.446 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.447 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.448 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.449 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.450 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj8", 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 28 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj8", 2024-09-10T21:57:00.451 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2381834917, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'28", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'8", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4301.0:1", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 28, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.131158+0000", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.131700+0000", 2024-09-10T21:57:00.452 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.453 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.454 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.455 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.456 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj9", 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 30 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:57:00.457 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj9", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 3833113727, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'30", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'9", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4304.0:1", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 30, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:55:52.264696+0000", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:55:52.265225+0000", 2024-09-10T21:57:00.458 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xc32219df", 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.459 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:57:00.460 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:57:00.461 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: "size": "????", 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:57:00.462 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:57:00.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='490412 ' 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='489848 ' 2024-09-10T21:57:00.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-10T21:57:00.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2024-09-10T21:57:00.467 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2024-09-10T21:57:00.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:00.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-09-10T21:57:00.590 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap1 2024-09-10T21:57:00.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:00.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-09-10T21:57:00.804 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap2 2024-09-10T21:57:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:00.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-09-10T21:57:00.997 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap3 2024-09-10T21:57:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-09-10T21:57:01.180 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap4 2024-09-10T21:57:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:01.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-09-10T21:57:01.467 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap5 2024-09-10T21:57:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:01.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-09-10T21:57:01.743 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap6 2024-09-10T21:57:01.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-10T21:57:01.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-09-10T21:57:01.935 INFO:tasks.workunit.client.0.smithi181.stdout:removed pool test snap snap7 2024-09-10T21:57:01.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2024-09-10T21:57:06.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2024-09-10T21:57:06.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2024-09-10T21:57:06.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2024-09-10T21:57:07.219 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:57:07.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2024-09-10T21:57:07.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2024-09-10T21:57:07.501 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:57:07.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2024-09-10T21:57:07.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-10T21:57:07.786 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T21:57:07.786 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-09-10T21:57:01.979395+0000 23'56 37:132 [1,0] 1 [1,0] 1 23'56 2024-09-10T21:56:55.433612+0000 0'0 2024-09-10T21:55:42.887008+0000 7 1 periodic scrub scheduled @ 2024-09-12T07:57:52.797919+0000 33 0 2024-09-10T21:57:07.786 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T21:57:07.786 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T21:57:07.786 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 490412 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 489848 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:57:07.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:57:13.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:57:13.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:57:13.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-10T21:57:13.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2024-09-10T21:57:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2024-09-10T21:57:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2024-09-10T21:57:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-10T21:57:13.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-10T21:57:13.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:57:13.136 INFO:tasks.workunit.client.0.smithi181.stdout:TEST PASSED 2024-09-10T21:57:13.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-10T21:57:13.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-10T21:57:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-10T21:57:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2024-09-10T21:57:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:57:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2024-09-10T21:57:13.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:57:13.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:57:13.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:57:13.142 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:57:13.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:57:13.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:57:13.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:57:13.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:57:13.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:57:13.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:57:13.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:57:13.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:57:13.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:57:13.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:57:13.148 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:57:13.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:57:13.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:57:13.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:57:13.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:57:13.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:57:13.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-10T21:57:13.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-10T21:57:13.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:57:13.167 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:57:13.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:57:13.168 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:57:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:57:13.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:57:13.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:57:13.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:57:13.171 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:57:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:57:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:57:13.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:57:13.172 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:57:13.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:57:13.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:57:13.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:57:13.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:57:13.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:57:13.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:57:13.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:57:13.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-10T21:57:13.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:57:13.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.178 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.455221 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-10T21:57:13.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2024-09-10T21:57:13.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.455221 2024-09-10T21:57:13.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-10T21:57:13.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-10T21:57:13.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:57:13.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:57:13.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:57:13.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-10T21:57:13.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.654 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.655 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:13.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-10T21:57:13.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:57:13.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:57:13.865 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:57:13.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get fsid 2024-09-10T21:57:13.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:57:13.951 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.455221/ceph-mon.a.asok 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:57:13.952 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.455221/ceph-mon.a.asok config get mon_host 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-10T21:57:14.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:57:14.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:14.873 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:14.874 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:57:14.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:57:14.910 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-10T21:57:14.911 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:57:14.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:57:14.917 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:57:14.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0e14b35-4838-4e85-b995-f947a7838647 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e0e14b35-4838-4e85-b995-f947a7838647' 2024-09-10T21:57:14.919 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 e0e14b35-4838-4e85-b995-f947a7838647 2024-09-10T21:57:14.920 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:57:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6wOBmLuIXOBAAgzgBs5iQC6zuEQA3RYy0dA== 2024-09-10T21:57:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6wOBmLuIXOBAAgzgBs5iQC6zuEQA3RYy0dA=="}' 2024-09-10T21:57:14.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0e14b35-4838-4e85-b995-f947a7838647 -i td/osd-scrub-snaps/0/new.json 2024-09-10T21:57:15.187 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:57:15.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-10T21:57:15.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6wOBmLuIXOBAAgzgBs5iQC6zuEQA3RYy0dA== --osd-uuid e0e14b35-4838-4e85-b995-f947a7838647 2024-09-10T21:57:15.238 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:15.237+0000 7f97cadc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:15.240 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:15.240+0000 7f97cadc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:15.241+0000 7f97cadc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:15.241+0000 7f97cadc5640 -1 bdev(0x560623fd3c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:57:15.242 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:15.241+0000 7f97cadc5640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-10T21:57:20.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-10T21:57:20.977 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:57:20.977 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:57:20.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:57:20.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:57:21.345 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:57:21.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:57:21.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:57:21.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:57:21.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:57:21.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:57:21.383 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:21.382+0000 7f737149c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:21.409 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:21.408+0000 7f737149c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:21.410 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:21.409+0000 7f737149c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:21.632 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:57:21.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:57:21.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:21.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:21.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:22.523 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:22.522+0000 7f737149c640 -1 Falling back to public interface 2024-09-10T21:57:22.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:22.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:22.963 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:57:22.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:57:22.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:22.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:23.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:24.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:24.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:24.255 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:57:24.255 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:57:24.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:24.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:24.342 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:24.341+0000 7f737149c640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:57:24.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:25.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:25.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:25.547 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:57:25.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:57:25.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:25.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:25.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:26.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:26.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:26.848 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:57:26.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:57:26.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:26.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:27.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:28.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:28.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:28.169 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:57:28.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:57:28.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:28.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:28.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:29.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:29.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:29.466 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:57:29.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:57:29.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:29.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:29.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:30.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:30.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:30.769 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:57:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:57:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:30.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:31.114 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:32.115 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:32.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:32.116 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:57:32.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:57:32.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:32.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:32.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:33.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:33.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:33.412 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:57:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:57:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:33.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:33.712 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:34.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:34.714 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:34.714 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:57:34.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:57:34.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:34.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:35.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:36.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:36.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:36.135 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:57:36.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:57:36.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:36.136 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:36.600 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:37.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:37.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:37.601 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:57:37.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:57:37.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:37.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:38.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:39.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:39.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:39.315 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:57:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:57:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:39.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:40.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:41.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:41.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:41.025 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:57:41.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:57:41.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:41.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:41.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:42.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:42.736 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:57:42.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:57:42.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:42.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:43.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:44.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:44.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:44.183 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:57:44.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:57:44.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:44.183 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:44.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:45.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:45.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:45.915 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:57:45.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:57:45.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:45.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:46.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:47.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:47.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:47.322 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:57:47.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:57:47.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:47.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:57:47.612 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1687044679,v1:127.0.0.1:6803/1687044679] [v2:127.0.0.1:6804/1687044679,v1:127.0.0.1:6805/1687044679] exists,up e0e14b35-4838-4e85-b995-f947a7838647 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:57:47.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:57:47.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:57:47.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=852da02c-1db4-40ba-8466-0f69ced8ffee 2024-09-10T21:57:47.619 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 852da02c-1db4-40ba-8466-0f69ced8ffee 2024-09-10T21:57:47.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 852da02c-1db4-40ba-8466-0f69ced8ffee' 2024-09-10T21:57:47.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:57:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbwOBmtw4/JhAAiSOXCvyrZd/wukQ1a/tGqQ== 2024-09-10T21:57:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbwOBmtw4/JhAAiSOXCvyrZd/wukQ1a/tGqQ=="}' 2024-09-10T21:57:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 852da02c-1db4-40ba-8466-0f69ced8ffee -i td/osd-scrub-snaps/1/new.json 2024-09-10T21:57:48.028 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:57:48.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-10T21:57:48.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbwOBmtw4/JhAAiSOXCvyrZd/wukQ1a/tGqQ== --osd-uuid 852da02c-1db4-40ba-8466-0f69ced8ffee 2024-09-10T21:57:48.080 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:48.078+0000 7f06ae811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:48.111 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:48.109+0000 7f06ae811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:48.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:48.110+0000 7f06ae811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:48.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:48.110+0000 7f06ae811640 -1 bdev(0x55ecc84c1c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:57:48.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:48.110+0000 7f06ae811640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-10T21:57:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-10T21:57:53.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:57:53.900 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T21:57:53.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T21:57:53.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:57:54.269 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:57:54.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T21:57:54.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:57:54.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:57:54.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:57:54.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:57:54.308 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:54.306+0000 7fc359a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:54.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:54.330+0000 7fc359a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:54.333 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:54.331+0000 7fc359a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:57:54.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T21:57:54.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:57:54.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:54.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:57:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:55.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:55.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:55.887 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:57:55.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:57:55.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:55.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:57:55.937 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:55.935+0000 7fc359a2f640 -1 Falling back to public interface 2024-09-10T21:57:56.184 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:57.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:57.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:57.186 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:57:57.186 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:57:57.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:57.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:57:57.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:57.741 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:57:57.740+0000 7fc359a2f640 -1 osd.1 0 log_to_monitors true 2024-09-10T21:57:58.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:58.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:58.480 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:57:58.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:57:58.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:58.481 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:57:58.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:57:59.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:57:59.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:57:59.775 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T21:57:59.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T21:57:59.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:57:59.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:00.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:01.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:01.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:01.077 INFO:tasks.workunit.client.0.smithi181.stdout:5 2024-09-10T21:58:01.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T21:58:01.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:01.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:01.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:02.471 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T21:58:02.472 INFO:tasks.workunit.client.0.smithi181.stdout:6 2024-09-10T21:58:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:02.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:02.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:03.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:03.818 INFO:tasks.workunit.client.0.smithi181.stdout:7 2024-09-10T21:58:03.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:03.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T21:58:03.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:03.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:04.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:05.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:05.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:05.119 INFO:tasks.workunit.client.0.smithi181.stdout:8 2024-09-10T21:58:05.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T21:58:05.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:05.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:05.463 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:06.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:06.465 INFO:tasks.workunit.client.0.smithi181.stdout:9 2024-09-10T21:58:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T21:58:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:06.466 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:06.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:07.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:07.851 INFO:tasks.workunit.client.0.smithi181.stdout:10 2024-09-10T21:58:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T21:58:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:07.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:08.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:09.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:09.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:09.152 INFO:tasks.workunit.client.0.smithi181.stdout:11 2024-09-10T21:58:09.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T21:58:09.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:09.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:09.477 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:10.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:10.480 INFO:tasks.workunit.client.0.smithi181.stdout:12 2024-09-10T21:58:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T21:58:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:10.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:10.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:11.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:11.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:11.791 INFO:tasks.workunit.client.0.smithi181.stdout:13 2024-09-10T21:58:11.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T21:58:11.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:11.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:12.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:13.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:13.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:13.361 INFO:tasks.workunit.client.0.smithi181.stdout:14 2024-09-10T21:58:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T21:58:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:13.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:14.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:15.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:15.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:15.023 INFO:tasks.workunit.client.0.smithi181.stdout:15 2024-09-10T21:58:15.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T21:58:15.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:15.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:15.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:16.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:16.574 INFO:tasks.workunit.client.0.smithi181.stdout:16 2024-09-10T21:58:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T21:58:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:16.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:16.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:17.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:17.922 INFO:tasks.workunit.client.0.smithi181.stdout:17 2024-09-10T21:58:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T21:58:17.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:17.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:18.757 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:19.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:19.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:19.759 INFO:tasks.workunit.client.0.smithi181.stdout:18 2024-09-10T21:58:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T21:58:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:19.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:20.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:58:21.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:58:21.252 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:58:21.253 INFO:tasks.workunit.client.0.smithi181.stdout:19 2024-09-10T21:58:21.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-10T21:58:21.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:58:21.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:58:21.552 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3934005793,v1:127.0.0.1:6811/3934005793] [v2:127.0.0.1:6812/3934005793,v1:127.0.0.1:6813/3934005793] exists,up 852da02c-1db4-40ba-8466-0f69ced8ffee 2024-09-10T21:58:21.552 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:58:21.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:58:21.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:58:21.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-10T21:58:22.170 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T21:58:22.182 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-10T21:58:22.680 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T21:58:22.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-10T21:58:22.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T21:58:23.132 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T21:58:23.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T21:58:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-10T21:58:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:58:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:58:24.145 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:58:24.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:58:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:58:24.146 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:58:24.146 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:58:24.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:58:24.147 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:58:24.246 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:58:24.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:58:24.533 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:58:24.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:58:24.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:58:24.534 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:58:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-10T21:58:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-10T21:58:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-10T21:58:24.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:58:24.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:58:24.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-10T21:58:24.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-10T21:58:24.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-09-10T21:58:24.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:58:24.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-10T21:58:24.804 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:58:24.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:58:24.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-10T21:58:24.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:58:24.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-10T21:58:24.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-10T21:58:24.807 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836493 2024-09-10T21:58:24.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:58:25.121 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-10T21:58:25.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:58:25.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-10T21:58:25.122 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:58:25.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:58:25.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-10T21:58:25.124 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:58:25.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-10T21:58:25.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-10T21:58:25.125 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672965 2024-09-10T21:58:25.125 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:58:25.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-10T21:58:25.421 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:58:25.421 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:58:25.422 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:58:25.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:58:25.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:58:25.776 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:58:26.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:58:26.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:58:26.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:58:26.058 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:58:26.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:58:26.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:58:26.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:58:26.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-10T21:58:26.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-10T21:58:26.412 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-10T21:58:26.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-10T21:58:26.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.455221 bs=1032 count=1 2024-09-10T21:58:26.700 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:58:26.700 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:58:26.700 INFO:tasks.workunit.client.0.smithi181.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5454e-05 s, 22.7 MB/s 2024-09-10T21:58:26.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-10T21:58:26.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:26.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.455221 2024-09-10T21:58:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:26.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.455221 2024-09-10T21:58:27.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.455221 2024-09-10T21:58:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.455221 2024-09-10T21:58:27.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.455221 2024-09-10T21:58:27.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.514 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.455221 2024-09-10T21:58:27.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.455221 2024-09-10T21:58:27.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.455221 2024-09-10T21:58:27.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:27.924 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.455221 2024-09-10T21:58:28.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.455221 2024-09-10T21:58:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.209 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.455221 2024-09-10T21:58:28.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.455221 2024-09-10T21:58:28.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.455221 2024-09-10T21:58:28.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.455221 2024-09-10T21:58:28.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.455221 2024-09-10T21:58:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-10T21:58:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.455221 2024-09-10T21:58:29.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-10T21:58:29.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T21:58:29.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:58:29.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T21:58:29.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:58:29.328 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T21:58:29.329 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T21:58:29.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T21:58:29.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-10T21:58:29.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T21:58:29.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.455221 '$replica' 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.455221 0 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.455221 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-10T21:58:29.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-10T21:58:30.181 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap1 2024-09-10T21:58:30.183 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=1 2024-09-10T21:58:30.184 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T21:58:30.184 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T21:58:30.184 INFO:tasks.workunit.client.0.smithi181.stderr:256 bytes copied, 6.3879e-05 s, 4.0 MB/s 2024-09-10T21:58:30.184 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.455221 2024-09-10T21:58:30.417 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:58:30.559 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:58:30.701 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-10T21:58:30.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:30.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.455221 2024-09-10T21:58:30.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:30.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.455221 2024-09-10T21:58:31.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.455221 2024-09-10T21:58:31.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.455221 2024-09-10T21:58:31.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.455221 2024-09-10T21:58:31.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.413 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.455221 2024-09-10T21:58:31.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.555 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.455221 2024-09-10T21:58:31.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.455221 2024-09-10T21:58:31.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-10T21:58:31.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.455221 2024-09-10T21:58:31.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-10T21:58:31.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-10T21:58:32.149 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap2 2024-09-10T21:58:32.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=2 2024-09-10T21:58:32.152 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records in 2024-09-10T21:58:32.152 INFO:tasks.workunit.client.0.smithi181.stderr:2+0 records out 2024-09-10T21:58:32.152 INFO:tasks.workunit.client.0.smithi181.stderr:512 bytes copied, 6.5005e-05 s, 7.9 MB/s 2024-09-10T21:58:32.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:58:32.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-10T21:58:32.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-10T21:58:32.526 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap3 2024-09-10T21:58:32.527 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=3 2024-09-10T21:58:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records in 2024-09-10T21:58:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:3+0 records out 2024-09-10T21:58:32.528 INFO:tasks.workunit.client.0.smithi181.stderr:768 bytes copied, 7.033e-05 s, 10.9 MB/s 2024-09-10T21:58:32.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.455221 2024-09-10T21:58:32.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-10T21:58:32.828 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-10T21:58:33.060 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap4 2024-09-10T21:58:33.062 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=4 2024-09-10T21:58:33.063 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records in 2024-09-10T21:58:33.063 INFO:tasks.workunit.client.0.smithi181.stderr:4+0 records out 2024-09-10T21:58:33.063 INFO:tasks.workunit.client.0.smithi181.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.6913e-05 s, 15.3 MB/s 2024-09-10T21:58:33.063 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:58:33.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.455221 2024-09-10T21:58:33.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-10T21:58:33.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-10T21:58:33.631 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap5 2024-09-10T21:58:33.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-10T21:58:33.633 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-10T21:58:33.847 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap6 2024-09-10T21:58:33.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=6 2024-09-10T21:58:33.850 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records in 2024-09-10T21:58:33.850 INFO:tasks.workunit.client.0.smithi181.stderr:6+0 records out 2024-09-10T21:58:33.850 INFO:tasks.workunit.client.0.smithi181.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.6036e-05 s, 20.2 MB/s 2024-09-10T21:58:33.850 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.455221 2024-09-10T21:58:34.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-10T21:58:34.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-10T21:58:34.242 INFO:tasks.workunit.client.0.smithi181.stdout:created pool test snap snap7 2024-09-10T21:58:34.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-10T21:58:34.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-10T21:58:34.595 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-10T21:58:34.729 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-10T21:58:34.730 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:58:34.730 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:58:34.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:58:34.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:58:34.730 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:58:35.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:58:35.036 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-10T21:58:35.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:35.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-10T21:58:37.107 INFO:tasks.workunit.client.0.smithi181.stdout:WARNING: only removing head with clones present 2024-09-10T21:58:37.108 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-10T21:58:37.626 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:58:37.626 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-10T21:58:38.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:38.216 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:58:39.353 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:2# 2024-09-10T21:58:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:58:39.877 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-10T21:58:40.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:40.478 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:40.479 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:58:41.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:58:41.549 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.550 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.551 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:41.552 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:58:41.553 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:41.553 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:58:42.564 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:c52c9666:::obj5:1# 2024-09-10T21:58:43.089 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:58:44.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:58:44.153 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:44.154 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.155 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:58:44.156 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:44.157 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:58:44.157 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-10T21:58:44.157 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-10T21:58:44.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:44.758 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=18 2024-09-10T21:58:44.759 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records in 2024-09-10T21:58:44.759 INFO:tasks.workunit.client.0.smithi181.stderr:18+0 records out 2024-09-10T21:58:44.759 INFO:tasks.workunit.client.0.smithi181.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000117538 s, 39.2 MB/s 2024-09-10T21:58:44.759 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:58:46.231 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-10T21:58:46.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:46.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=15 2024-09-10T21:58:46.827 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records in 2024-09-10T21:58:46.827 INFO:tasks.workunit.client.0.smithi181.stderr:15+0 records out 2024-09-10T21:58:46.827 INFO:tasks.workunit.client.0.smithi181.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000112406 s, 34.2 MB/s 2024-09-10T21:58:46.827 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:58:48.326 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-10T21:58:48.326 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-10T21:58:48.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:48.925 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:58:49.992 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-10T21:58:50.757 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:58:51.826 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:51.827 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:51.828 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:51.829 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-10T21:58:51.830 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-10T21:58:52.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:52.415 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-10T21:58:53.455 INFO:tasks.workunit.client.0.smithi181.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-10T21:58:53.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-10T21:58:55.045 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-10T21:58:55.046 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-10T21:58:55.046 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-10T21:58:55.046 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-10T21:58:55.047 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:55.048 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:55.049 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-10T21:58:55.049 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:55.049 INFO:tasks.workunit.client.0.smithi181.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-10T21:58:55.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-10T21:58:55.049 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-10T21:58:55.050 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-10T21:58:55.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:55.640 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-10T21:58:57.105 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:57.105 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-10T21:58:57.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:57.106 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.455221 bs=256 count=7 2024-09-10T21:58:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records in 2024-09-10T21:58:57.107 INFO:tasks.workunit.client.0.smithi181.stderr:7+0 records out 2024-09-10T21:58:57.108 INFO:tasks.workunit.client.0.smithi181.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.4827e-05 s, 18.9 MB/s 2024-09-10T21:58:57.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.455221 2024-09-10T21:58:58.579 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-10T21:58:58.905 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:58:59.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:58:59.169 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-10T21:59:00.648 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-10T21:59:00.972 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:01.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:01.244 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-10T21:59:02.964 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-10T21:59:03.298 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:03.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:03.569 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-10T21:59:05.038 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-10T21:59:05.364 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:05.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:05.632 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-10T21:59:07.073 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-10T21:59:07.398 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:07.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:07.667 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-10T21:59:09.358 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-10T21:59:09.689 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:09.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:09.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-10T21:59:11.390 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-10T21:59:11.714 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:11.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:11.982 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-10T21:59:13.487 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-10T21:59:13.825 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:14.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:14.096 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-10T21:59:15.657 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-10T21:59:15.988 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:16.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:16.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-10T21:59:18.044 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-10T21:59:18.045 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-10T21:59:18.378 INFO:tasks.workunit.client.0.smithi181.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-10T21:59:18.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-10T21:59:18.645 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-10T21:59:20.097 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-10T21:59:20.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-10T21:59:20.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.455221 2024-09-10T21:59:20.100 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-10T21:59:20.101 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T21:59:20.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:59:20.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:59:20.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-10T21:59:20.109 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:59:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T21:59:20.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:59:20.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-10T21:59:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T21:59:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:59:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:59:20.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:59:20.149 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:20.147+0000 7fe270433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:20.151 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:20.150+0000 7fe270433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:20.152 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:20.151+0000 7fe270433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:59:20.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:20.404 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:59:20.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:59:20.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:20.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:20.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:21.517 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:21.516+0000 7fe270433640 -1 Falling back to public interface 2024-09-10T21:59:21.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:21.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:21.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:59:21.734 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:59:21.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:21.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:22.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:23.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:23.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:23.031 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:59:23.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:59:23.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:23.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:23.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:23.053+0000 7fe270433640 -1 osd.0 23 log_to_monitors true 2024-09-10T21:59:23.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:24.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:24.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:24.368 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:59:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:59:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:24.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:24.660 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2187166803,v1:127.0.0.1:6803/2187166803] [v2:127.0.0.1:6804/2187166803,v1:127.0.0.1:6805/2187166803] exists,up e0e14b35-4838-4e85-b995-f947a7838647 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-10T21:59:24.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T21:59:24.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:59:24.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T21:59:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T21:59:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-10T21:59:24.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T21:59:24.667 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.455221/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T21:59:24.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T21:59:24.703 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:24.702+0000 7f8332dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:24.717 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:24.716+0000 7f8332dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:24.718 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:24.717+0000 7f8332dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:24.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:24.958 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:59:24.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:59:24.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:24.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:59:25.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:25.827 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:25.826+0000 7f8332dbd640 -1 Falling back to public interface 2024-09-10T21:59:26.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:26.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:26.272 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:59:26.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:59:26.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:26.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:59:26.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:27.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:27.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:27.570 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:59:27.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:59:27.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:27.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:59:27.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:27.877 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:27.875+0000 7f8332dbd640 -1 osd.1 23 log_to_monitors true 2024-09-10T21:59:28.862 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:28.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:59:28.863 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:59:28.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:28.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T21:59:29.160 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/2684063834,v1:127.0.0.1:6811/2684063834] [v2:127.0.0.1:6812/2684063834,v1:127.0.0.1:6813/2684063834] exists,up 852da02c-1db4-40ba-8466-0f69ced8ffee 2024-09-10T21:59:29.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:59:29.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:59:29.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:59:29.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-10T21:59:29.274 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:59:29.274 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T21:59:29.274 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.287 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:59:29.287 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-10T21:59:29.287 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-10T21:59:29.425 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:59:29.425 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:59:29.425 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.439 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:59:29.439 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:59:29.439 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-10T21:59:29.575 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:59:29.576 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:59:29.576 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.589 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:59:29.589 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-10T21:59:29.589 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-10T21:59:29.717 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:59:29.718 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-10T21:59:29.718 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.731 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:59:29.731 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-10T21:59:29.731 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.740 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-10T21:59:29.877 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T21:59:29.878 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-10T21:59:29.878 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.891 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T21:59:29.891 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-10T21:59:29.891 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:29.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-10T21:59:29.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T21:59:29.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T21:59:29.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T21:59:29.900 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T21:59:29.901 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T21:59:29.901 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T21:59:29.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T21:59:29.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T21:59:29.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T21:59:29.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T21:59:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T21:59:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T21:59:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T21:59:29.990 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T21:59:29.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T21:59:30.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T21:59:30.276 INFO:tasks.workunit.client.0.smithi181.stderr:1' 2024-09-10T21:59:30.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T21:59:30.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:59:30.277 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T21:59:30.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-10T21:59:30.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-10T21:59:30.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699' 2024-09-10T21:59:30.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T21:59:30.430 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T21:59:30.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-10T21:59:30.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-10T21:59:30.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-10T21:59:30.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:59:30.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149699 2024-09-10T21:59:30.543 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:59:30.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T21:59:30.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149699 2024-09-10T21:59:30.544 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:59:30.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-10T21:59:30.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-10T21:59:30.546 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 111669149699 2024-09-10T21:59:30.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:59:30.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149699 2024-09-10T21:59:30.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T21:59:31.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T21:59:31.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T21:59:32.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-10T21:59:32.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T21:59:32.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-10T21:59:32.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T21:59:32.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T21:59:32.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-10T21:59:32.133 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T21:59:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-10T21:59:32.134 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 124554051586 2024-09-10T21:59:32.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-10T21:59:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T21:59:32.464 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-10T21:59:32.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T21:59:32.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:59:32.464 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T21:59:32.865 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T21:59:32.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T21:59:32.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T21:59:33.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T21:59:33.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T21:59:33.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T21:59:33.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T21:59:33.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T21:59:33.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:33.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:33.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:33.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-10T21:59:34.086 INFO:tasks.workunit.client.0.smithi181.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-10T21:59:34.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:34.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T21:59:34.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:34.099 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:34.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:58:23.081320+0000 '>' 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:34.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:59:35.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:59:35.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:59:35.382 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:59:35.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:35.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:35.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:35.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:35.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:58:23.081320+0000 '>' 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:35.677 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:59:36.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:59:36.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:59:36.678 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:59:36.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:36.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:36.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:36.679 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:36.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:58:23.081320+0000 '>' 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:36.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:59:37.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:59:37.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:59:37.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:59:37.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:37.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:37.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:37.976 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:58:23.081320+0000 '>' 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:38.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T21:59:39.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T21:59:39.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-10T21:59:39.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T21:59:39.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T21:59:39.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T21:59:39.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T21:59:39.280 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T21:59:39.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T21:59:35.536782+0000 '>' 2024-09-10T21:58:23.081320+0000 2024-09-10T21:59:39.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T21:59:39.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:39.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-10T21:59:39.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2024-09-10T21:59:39.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:59:39.577 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-10T21:59:39.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2024-09-10T21:59:39.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-10T21:59:39.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-10T21:59:39.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-10T21:59:39.618 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-10T21:59:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-10T21:59:39.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-10T21:59:39.655 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch": 29, 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "inconsistents": [ 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 5 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.656 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 1, 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'19", 2024-09-10T21:59:39.657 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'5", 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4241.0:1", 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 5, 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:27.424185+0000", 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:27.424827+0000", 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.658 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x7ea3985a", 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.659 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.660 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 2, 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 20 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.661 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 2, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "18'41", 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'20", 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4289.0:1", 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 20, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:30.453030+0000", 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:30.453534+0000", 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.662 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.663 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.664 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 4, 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 42 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch" 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.665 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 4, 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "20'45", 2024-09-10T21:59:39.666 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "18'42", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4325.0:1", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 42, 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512, 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:32.187296+0000", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:32.197461+0000", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x4c92d67b", 2024-09-10T21:59:39.667 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.668 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 4608, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "object_info": { 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj5", 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 4, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1718170787, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.669 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "20'45", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "18'42", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4325.0:1", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 42, 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512, 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:32.187296+0000", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:32.197461+0000", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.670 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x4c92d67b", 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.671 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 512 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj4", 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.672 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 4 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj4", 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": 7, 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2826278768, 2024-09-10T21:59:39.673 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "23'51", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'4", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4238.0:1", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 4, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:27.282020+0000", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:27.282653+0000", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.674 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x7ea3985a", 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.675 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032 2024-09-10T21:59:39.676 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj5", 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 0 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "missing", 2024-09-10T21:59:39.677 INFO:tasks.workunit.client.0.smithi181.stdout: "info_missing" 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "info_missing" 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1792 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.678 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj1", 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 18 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.679 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj1", 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1828249343, 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'18", 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'1", 2024-09-10T21:59:39.680 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4286.0:1", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 18, 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:30.219913+0000", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:30.306078+0000", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.681 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: "missing" 2024-09-10T21:59:39.682 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj10", 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.683 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 32 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj10", 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 718195851, 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.684 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'32", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'10", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4307.0:1", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 32, 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.306511+0000", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.307153+0000", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.685 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.686 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "????", 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.687 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.688 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj11", 2024-09-10T21:59:39.689 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 34 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj11", 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.690 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 693400951, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'34", 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'11", 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4310.0:1", 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 34, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.448318+0000", 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.448924+0000", 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.691 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.692 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.693 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.694 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj13", 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 38 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.695 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj13", 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2087409765, 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.696 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'38", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'13", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4316.0:1", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 38, 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.724458+0000", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.725021+0000", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.697 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.698 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]" 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.699 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.700 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj14", 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 40 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.701 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj14", 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2484217095, 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'40", 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'14", 2024-09-10T21:59:39.702 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4319.0:1", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 40, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.900104+0000", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.900652+0000", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.703 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.704 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1033, 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.705 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.706 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj15", 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 15 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.707 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_corrupted" 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj15", 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 612772309, 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.708 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "16'15", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "0'0", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4271.0:1", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 15, 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:28.796062+0000", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:28.796693+0000", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x7ea3985a", 2024-09-10T21:59:39.709 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_corrupted" 2024-09-10T21:59:39.710 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.711 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj2", 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 56 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_missing" 2024-09-10T21:59:39.712 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj2", 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1058988552, 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "23'56", 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "20'48", 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4361.0:1", 2024-09-10T21:59:39.713 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 56, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:34.629611+0000", 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:34.630263+0000", 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "whiteout", 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty" 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0xffffffff", 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.714 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_missing" 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.715 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 0, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 7, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 4, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.716 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 4, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 3, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 2, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 7, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1024, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 7, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 6, 2024-09-10T21:59:39.717 INFO:tasks.workunit.client.0.smithi181.stdout: 5 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj3", 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 44 2024-09-10T21:59:39.718 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch" 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [ 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj3", 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1643547569, 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.719 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19'44", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'22", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4331.0:1", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 44, 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768, 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:32.564235+0000", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:32.582969+0000", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.720 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x69992f6b", 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.721 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "size_mismatch_info", 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "obj_size_info_mismatch" 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 3840, 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "object_info": { 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj3", 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1643547569, 2024-09-10T21:59:39.722 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "19'44", 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "17'22", 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4331.0:1", 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 44, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:32.564235+0000", 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:32.582969+0000", 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.723 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x69992f6b", 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.724 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 768 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj6", 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.725 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 24 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj6", 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2202164420, 2024-09-10T21:59:39.726 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'24", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'6", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4295.0:1", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 24, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:30.738051+0000", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:30.738600+0000", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.727 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.728 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.729 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.730 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj7", 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 26 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.731 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj7", 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 1552453721, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'26", 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'7", 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4298.0:1", 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 26, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.732 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:30.879391+0000", 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:30.879956+0000", 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.733 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [] 2024-09-10T21:59:39.734 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.735 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj8", 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.736 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 28 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj8", 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 2381834917, 2024-09-10T21:59:39.737 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'28", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'8", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4301.0:1", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 28, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.038292+0000", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.038924+0000", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.738 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.739 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 0, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.740 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.741 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: "object": { 2024-09-10T21:59:39.742 INFO:tasks.workunit.client.0.smithi181.stdout: "name": "obj9", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "nspace": "", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "locator": "", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": "head", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "version": 30 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [ 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset_inconsistency" 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "union_shard_errors": [], 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "selected_object_info": { 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": { 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "oid": "obj9", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "key": "", 2024-09-10T21:59:39.743 INFO:tasks.workunit.client.0.smithi181.stdout: "snapid": -2, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "hash": 3833113727, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "max": 0, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "pool": 1, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "namespace": "" 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "version": "17'30", 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "prior_version": "16'9", 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "last_reqid": "client.4304.0:1", 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "user_version": 30, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "mtime": "2024-09-10T21:58:31.164952+0000", 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "local_mtime": "2024-09-10T21:58:31.165484+0000", 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "lost": 0, 2024-09-10T21:59:39.744 INFO:tasks.workunit.client.0.smithi181.stdout: "flags": [ 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "dirty", 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest" 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_seq": 0, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "truncate_size": 0, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "data_digest": "0x6461c389", 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "omap_digest": "0xffffffff", 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_object_size": 0, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "expected_write_size": 0, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "alloc_hint_flags": 0, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "manifest": { 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "type": 0 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.745 INFO:tasks.workunit.client.0.smithi181.stdout: "watchers": {} 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "shards": [ 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 0, 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": false, 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.746 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: "size": "????", 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.747 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "osd": 1, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "primary": true, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "errors": [], 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 256, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "snapset": { 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "seq": 1, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "clones": [ 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "snap": 1, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "size": 1032, 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "overlap": "[]", 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: "snaps": [ 2024-09-10T21:59:39.748 INFO:tasks.workunit.client.0.smithi181.stdout: 1 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T21:59:39.749 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T21:59:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-10T21:59:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2024-09-10T21:59:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2024-09-10T21:59:39.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-10T21:59:39.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='509918 ' 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='509354 ' 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2024-09-10T21:59:39.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-10T21:59:40.015 INFO:tasks.workunit.client.0.smithi181.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T21:59:40.015 INFO:tasks.workunit.client.0.smithi181.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-09-10T21:59:35.536854+0000 23'56 30:103 [1,0] 1 [1,0] 1 23'56 2024-09-10T21:59:35.536782+0000 0'0 2024-09-10T21:58:23.081320+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:12:53.982975+0000 36 0 2024-09-10T21:59:40.016 INFO:tasks.workunit.client.0.smithi181.stdout: 2024-09-10T21:59:40.016 INFO:tasks.workunit.client.0.smithi181.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T21:59:40.016 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 509918 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 509354 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-10T21:59:40.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:59:40.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:59:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:59:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:59:40.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:59:45.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-10T21:59:45.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.354 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.356 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-10T21:59:45.360 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-09-10T21:59:45.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-10T21:59:45.363 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2024-09-10T21:59:45.363 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2024-09-10T21:59:45.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2024-09-10T21:59:45.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-10T21:59:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-10T21:59:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-10T21:59:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-10T21:59:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-10T21:59:45.368 INFO:tasks.workunit.client.0.smithi181.stdout:TEST PASSED 2024-09-10T21:59:45.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-10T21:59:45.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2024-09-10T21:59:45.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=e760bcc6-6dd9-4d10-838f-2c26dceecb48 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-10T21:59:45.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2024-09-10T21:59:45.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-10T21:59:45.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:59:45.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:59:45.370 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:59:45.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:59:45.371 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:59:45.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:59:45.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:59:45.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:59:45.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:59:45.376 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:59:45.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:59:45.378 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:59:45.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:59:45.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:59:45.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:59:45.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:45.394 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:59:45.394 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-snaps 0 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-10T21:59:45.396 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:59:45.400 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:59:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:59:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:59:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:59:45.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:59:45.401 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:59:45.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:59:45.402 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:59:45.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:59:45.404 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:59:45.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:59:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:59:45.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:59:45.408 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:59:45.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T21:59:45.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-10T21:59:45.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:59:45.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:45.409 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.455221 2024-09-10T21:59:45.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.455221 2024-09-10T21:59:45.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:59:45.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:59:45.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T21:59:45.423 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T21:59:45.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T21:59:45.497 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-09-10T21:59:45.497 DEBUG:teuthology.orchestra.run.smithi181:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-09-10T21:59:45.565 INFO:tasks.workunit.client.0.smithi181.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T21:59:45.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-test 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-09-10T21:59:45.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-09-10T21:59:45.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none ' 2024-09-10T21:59:45.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-09-10T21:59:45.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-10T21:59:45.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-09-10T21:59:45.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_dump_scrub_schedule 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_interval_changes 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_just_deep_scrubs 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_pg_dump_objects_scrubbed 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_abort 2024-09-10T21:59:45.575 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_permit_time 2024-09-10T21:59:45.576 INFO:tasks.workunit.client.0.smithi181.stderr:TEST_scrub_test' 2024-09-10T21:59:45.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-10T21:59:45.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-09-10T21:59:45.576 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-09-10T21:59:45.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-10T21:59:45.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-10T21:59:45.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-10T21:59:45.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T21:59:45.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T21:59:45.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T21:59:45.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T21:59:45.579 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T21:59:45.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T21:59:45.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T21:59:45.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T21:59:45.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T21:59:45.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T21:59:45.581 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T21:59:45.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T21:59:45.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T21:59:45.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-10T21:59:45.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T21:59:45.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:45.584 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:45.584 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.511250 2024-09-10T21:59:45.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T21:59:45.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T21:59:45.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-10T21:59:45.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T21:59:45.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:45.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:45.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.511250 2024-09-10T21:59:45.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T21:59:45.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T21:59:45.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T21:59:45.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T21:59:45.588 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-10T21:59:45.589 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-09-10T21:59:45.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local dir=td/osd-scrub-test 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local poolname=test 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local OSDS=3 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: local objects=1000 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: local type=deep-scrub 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: TESTDATA=testdata.511250 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:307: _scrub_abort: test deep-scrub = scrub 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:312: _scrub_abort: stopscrub=nodeep-scrub 2024-09-10T21:59:45.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: check=nodeep_scrub 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-10T21:59:45.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:46.067 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:46.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-10T21:59:46.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T21:59:46.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T21:59:46.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:59:46.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:59:46.260 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T21:59:46.261 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T21:59:46.261 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:59:46.261 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:59:46.261 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.511250/ceph-mon.a.asok 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:59:46.262 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.511250/ceph-mon.a.asok config get fsid 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T21:59:46.348 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.511250/ceph-mon.a.asok 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T21:59:46.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.511250/ceph-mon.a.asok config get mon_host 2024-09-10T21:59:46.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-10T21:59:46.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-10T21:59:46.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T21:59:46.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T21:59:46.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T21:59:46.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-10T21:59:46.429 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T21:59:47.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:47.236 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:59:47.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T21:59:47.265 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: expr 3 - 1 2024-09-10T21:59:47.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: seq 0 2 2024-09-10T21:59:47.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:59:47.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T21:59:47.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T21:59:47.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:59:47.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:59:47.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=099846f8-4b24-4ad6-ba44-c26028ca3212 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 099846f8-4b24-4ad6-ba44-c26028ca3212' 2024-09-10T21:59:47.273 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 099846f8-4b24-4ad6-ba44-c26028ca3212 2024-09-10T21:59:47.274 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:59:47.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTweBmoYILERAA7JaO0PXZWhN1tqnI8wSgnA== 2024-09-10T21:59:47.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTweBmoYILERAA7JaO0PXZWhN1tqnI8wSgnA=="}' 2024-09-10T21:59:47.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 099846f8-4b24-4ad6-ba44-c26028ca3212 -i td/osd-scrub-test/0/new.json 2024-09-10T21:59:47.491 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:59:47.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-10T21:59:47.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBTweBmoYILERAA7JaO0PXZWhN1tqnI8wSgnA== --osd-uuid 099846f8-4b24-4ad6-ba44-c26028ca3212 2024-09-10T21:59:47.534 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:47.532+0000 7f106e345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:47.542 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:47.541+0000 7f106e345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:47.543 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:47.542+0000 7f106e345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:47.543+0000 7f106e345640 -1 bdev(0x560594e9dc00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-10T21:59:47.544 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:47.543+0000 7f106e345640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-10T21:59:53.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-10T21:59:53.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T21:59:53.030 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T21:59:53.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T21:59:53.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T21:59:53.397 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T21:59:53.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T21:59:53.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T21:59:53.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T21:59:53.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T21:59:53.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T21:59:53.426 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:53.425+0000 7f351b334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:53.455 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:53.454+0000 7f351b334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:53.456 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:53.455+0000 7f351b334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:53.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T21:59:53.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T21:59:53.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T21:59:53.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T21:59:53.673 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T21:59:53.673 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T21:59:53.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:53.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T21:59:53.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:53.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:53.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:54.804 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:54.803+0000 7f351b334640 -1 Falling back to public interface 2024-09-10T21:59:54.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T21:59:54.988 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:59:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:55.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:56.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:56.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:56.270 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T21:59:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T21:59:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:56.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:56.370 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:56.369+0000 7f351b334640 -1 osd.0 0 log_to_monitors true 2024-09-10T21:59:56.558 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T21:59:57.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T21:59:57.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T21:59:57.561 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T21:59:57.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T21:59:57.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T21:59:57.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T21:59:57.869 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/249243581,v1:127.0.0.1:6803/249243581] [v2:127.0.0.1:6804/249243581,v1:127.0.0.1:6805/249243581] exists,up 099846f8-4b24-4ad6-ba44-c26028ca3212 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T21:59:57.873 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T21:59:57.874 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T21:59:57.875 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-10T21:59:57.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-10T21:59:57.877 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T21:59:57.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8272a7c4-74b3-49e6-81e7-4453eddaf8cf 2024-09-10T21:59:57.877 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 8272a7c4-74b3-49e6-81e7-4453eddaf8cf 2024-09-10T21:59:57.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8272a7c4-74b3-49e6-81e7-4453eddaf8cf' 2024-09-10T21:59:57.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T21:59:57.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdweBmUcDrNBAAMJuJUzGmhHctU3M4OAOKZA== 2024-09-10T21:59:57.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdweBmUcDrNBAAMJuJUzGmhHctU3M4OAOKZA=="}' 2024-09-10T21:59:57.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8272a7c4-74b3-49e6-81e7-4453eddaf8cf -i td/osd-scrub-test/1/new.json 2024-09-10T21:59:58.248 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T21:59:58.259 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-10T21:59:58.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBdweBmUcDrNBAAMJuJUzGmhHctU3M4OAOKZA== --osd-uuid 8272a7c4-74b3-49e6-81e7-4453eddaf8cf 2024-09-10T21:59:58.291 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:58.290+0000 7f04650b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:58.330 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:58.329+0000 7f04650b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:58.331 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:58.330+0000 7f04650b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T21:59:58.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:58.331+0000 7f04650b0640 -1 bdev(0x555ce13b3c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-10T21:59:58.332 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T21:59:58.331+0000 7f04650b0640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-10T22:00:04.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-10T22:00:04.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T22:00:04.092 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T22:00:04.093 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T22:00:04.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T22:00:04.450 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T22:00:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T22:00:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T22:00:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T22:00:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T22:00:04.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T22:00:04.479 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:04.478+0000 7f40491bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:04.481 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:04.480+0000 7f40491bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:04.482 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:04.481+0000 7f40491bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:04.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:04.725 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:00:04.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T22:00:04.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:04.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:00:05.006 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:05.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:05.831+0000 7f40491bd640 -1 Falling back to public interface 2024-09-10T22:00:06.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:06.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:06.007 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:00:06.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T22:00:06.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:06.008 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:00:06.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:07.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:07.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:07.283 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:00:07.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T22:00:07.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:07.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:00:07.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:07.916 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:07.914+0000 7f40491bd640 -1 osd.1 0 log_to_monitors true 2024-09-10T22:00:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:08.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:08.610 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T22:00:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T22:00:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:08.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:00:08.895 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/195897178,v1:127.0.0.1:6811/195897178] [v2:127.0.0.1:6812/195897178,v1:127.0.0.1:6813/195897178] exists,up 8272a7c4-74b3-49e6-81e7-4453eddaf8cf 2024-09-10T22:00:08.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T22:00:08.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:00:08.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:00:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:00:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T22:00:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T22:00:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T22:00:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4dfee0db-860e-4082-98bb-73d5ad14bc51 2024-09-10T22:00:08.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4dfee0db-860e-4082-98bb-73d5ad14bc51' 2024-09-10T22:00:08.903 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 4dfee0db-860e-4082-98bb-73d5ad14bc51 2024-09-10T22:00:08.903 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T22:00:08.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoweBmgBeGNhAAR1NW0JXh/rFCiXvZgeOwjw== 2024-09-10T22:00:08.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoweBmgBeGNhAAR1NW0JXh/rFCiXvZgeOwjw=="}' 2024-09-10T22:00:08.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4dfee0db-860e-4082-98bb-73d5ad14bc51 -i td/osd-scrub-test/2/new.json 2024-09-10T22:00:09.360 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:00:09.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-10T22:00:09.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBoweBmgBeGNhAAR1NW0JXh/rFCiXvZgeOwjw== --osd-uuid 4dfee0db-860e-4082-98bb-73d5ad14bc51 2024-09-10T22:00:09.404 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:09.403+0000 7f40683bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:09.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:09.435+0000 7f40683bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:09.437 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:09.437+0000 7f40683bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:09.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:09.437+0000 7f40683bd640 -1 bdev(0x5653a2e16800 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-10T22:00:09.438 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:09.437+0000 7f40683bd640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-10T22:00:15.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-10T22:00:15.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T22:00:15.388 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T22:00:15.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T22:00:15.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T22:00:15.749 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T22:00:15.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T22:00:15.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-10T22:00:15.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T22:00:15.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T22:00:15.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T22:00:15.776 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:15.775+0000 7f5941c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:15.778 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:15.777+0000 7f5941c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:15.779 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:15.778+0000 7f5941c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T22:00:16.022 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:00:16.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:16.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:00:16.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:17.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:17.296 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:17.297 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:00:17.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T22:00:17.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:17.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:00:17.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:17.628 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:17.627+0000 7f5941c0d640 -1 Falling back to public interface 2024-09-10T22:00:18.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:18.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:18.582 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:00:18.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T22:00:18.582 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:18.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:00:18.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:19.854 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:00:19.851+0000 7f5941c0d640 -1 osd.2 0 log_to_monitors true 2024-09-10T22:00:19.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:19.865 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T22:00:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T22:00:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:19.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:00:20.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:00:21.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:00:21.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T22:00:21.171 INFO:tasks.workunit.client.0.smithi181.stdout:4 2024-09-10T22:00:21.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T22:00:21.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:00:21.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:00:21.458 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/662799037,v1:127.0.0.1:6819/662799037] [v2:127.0.0.1:6820/662799037,v1:127.0.0.1:6821/662799037] exists,up 4dfee0db-860e-4082-98bb-73d5ad14bc51 2024-09-10T22:00:21.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T22:00:21.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T22:00:21.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T22:00:21.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: create_pool test 1 1 2024-09-10T22:00:21.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T22:00:21.915 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T22:00:21.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T22:00:22.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:333: _scrub_abort: wait_for_clean 2024-09-10T22:00:22.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T22:00:22.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T22:00:22.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T22:00:22.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T22:00:22.930 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T22:00:22.930 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T22:00:22.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T22:00:22.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T22:00:22.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T22:00:23.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T22:00:23.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T22:00:23.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T22:00:23.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T22:00:23.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:00:23.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:00:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:00:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:00:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:00:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:00:23.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:00:23.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:00:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-10T22:00:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-10T22:00:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-10T22:00:23.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:00:23.456 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:00:23.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672964 2024-09-10T22:00:23.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672964 2024-09-10T22:00:23.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-09-10T22:00:23.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:00:23.556 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:00:23.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-10T22:00:23.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-10T22:00:23.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-09-10T22:00:23.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:00:23.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-10T22:00:23.689 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:00:23.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:00:23.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-10T22:00:23.690 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:00:23.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-10T22:00:23.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-10T22:00:23.692 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836487 2024-09-10T22:00:23.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:00:23.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-10T22:00:23.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:00:24.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:00:24.988 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:00:25.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-10T22:00:25.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:00:26.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T22:00:26.304 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:00:26.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-10T22:00:26.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:00:26.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672964 2024-09-10T22:00:26.586 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:00:26.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:00:26.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672964 2024-09-10T22:00:26.588 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:00:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672964 2024-09-10T22:00:26.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-10T22:00:26.589 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672964 2024-09-10T22:00:26.589 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:00:26.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-10T22:00:26.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:00:26.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-10T22:00:26.864 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:00:26.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:00:26.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-10T22:00:26.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:00:26.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-10T22:00:26.867 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509442 2024-09-10T22:00:26.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-10T22:00:26.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:00:27.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-10T22:00:27.150 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T22:00:27.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T22:00:27.151 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T22:00:27.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T22:00:27.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T22:00:27.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T22:00:27.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T22:00:27.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T22:00:27.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T22:00:27.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T22:00:27.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T22:00:27.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T22:00:27.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T22:00:27.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T22:00:27.775 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T22:00:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T22:00:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T22:00:28.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T22:00:28.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: ceph osd dump 2024-09-10T22:00:28.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-10T22:00:28.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: awk '{ print $2 }' 2024-09-10T22:00:28.418 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: poolid=1 2024-09-10T22:00:28.419 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: dd if=/dev/urandom of=testdata.511250 bs=1032 count=1 2024-09-10T22:00:28.419 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T22:00:28.419 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T22:00:28.419 INFO:tasks.workunit.client.0.smithi181.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.6023e-05 s, 22.4 MB/s 2024-09-10T22:00:28.419 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: seq 1 1000 2024-09-10T22:00:28.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:28.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1 testdata.511250 2024-09-10T22:00:28.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:28.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj2 testdata.511250 2024-09-10T22:00:28.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:28.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj3 testdata.511250 2024-09-10T22:00:28.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:28.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj4 testdata.511250 2024-09-10T22:00:29.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.061 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj5 testdata.511250 2024-09-10T22:00:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj6 testdata.511250 2024-09-10T22:00:29.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.346 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj7 testdata.511250 2024-09-10T22:00:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj8 testdata.511250 2024-09-10T22:00:29.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj9 testdata.511250 2024-09-10T22:00:29.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj10 testdata.511250 2024-09-10T22:00:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:29.941 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj11 testdata.511250 2024-09-10T22:00:30.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj12 testdata.511250 2024-09-10T22:00:30.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj13 testdata.511250 2024-09-10T22:00:30.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj14 testdata.511250 2024-09-10T22:00:30.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj15 testdata.511250 2024-09-10T22:00:30.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj16 testdata.511250 2024-09-10T22:00:30.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj17 testdata.511250 2024-09-10T22:00:30.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:30.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj18 testdata.511250 2024-09-10T22:00:31.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj19 testdata.511250 2024-09-10T22:00:31.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj20 testdata.511250 2024-09-10T22:00:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj21 testdata.511250 2024-09-10T22:00:31.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj22 testdata.511250 2024-09-10T22:00:31.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj23 testdata.511250 2024-09-10T22:00:31.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj24 testdata.511250 2024-09-10T22:00:31.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:31.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj25 testdata.511250 2024-09-10T22:00:32.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj26 testdata.511250 2024-09-10T22:00:32.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj27 testdata.511250 2024-09-10T22:00:32.420 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj28 testdata.511250 2024-09-10T22:00:32.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj29 testdata.511250 2024-09-10T22:00:32.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj30 testdata.511250 2024-09-10T22:00:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj31 testdata.511250 2024-09-10T22:00:32.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:32.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj32 testdata.511250 2024-09-10T22:00:33.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj33 testdata.511250 2024-09-10T22:00:33.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.191 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj34 testdata.511250 2024-09-10T22:00:33.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj35 testdata.511250 2024-09-10T22:00:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj36 testdata.511250 2024-09-10T22:00:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.652 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj37 testdata.511250 2024-09-10T22:00:33.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj38 testdata.511250 2024-09-10T22:00:33.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:33.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj39 testdata.511250 2024-09-10T22:00:34.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj40 testdata.511250 2024-09-10T22:00:34.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.164 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj41 testdata.511250 2024-09-10T22:00:34.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj42 testdata.511250 2024-09-10T22:00:34.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj43 testdata.511250 2024-09-10T22:00:34.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj44 testdata.511250 2024-09-10T22:00:34.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.711 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj45 testdata.511250 2024-09-10T22:00:34.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:34.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj46 testdata.511250 2024-09-10T22:00:35.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj47 testdata.511250 2024-09-10T22:00:35.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj48 testdata.511250 2024-09-10T22:00:35.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj49 testdata.511250 2024-09-10T22:00:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj50 testdata.511250 2024-09-10T22:00:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj51 testdata.511250 2024-09-10T22:00:35.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj52 testdata.511250 2024-09-10T22:00:35.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:35.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj53 testdata.511250 2024-09-10T22:00:36.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj54 testdata.511250 2024-09-10T22:00:36.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj55 testdata.511250 2024-09-10T22:00:36.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj56 testdata.511250 2024-09-10T22:00:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj57 testdata.511250 2024-09-10T22:00:36.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj58 testdata.511250 2024-09-10T22:00:36.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj59 testdata.511250 2024-09-10T22:00:36.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj60 testdata.511250 2024-09-10T22:00:36.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:36.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj61 testdata.511250 2024-09-10T22:00:37.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.107 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj62 testdata.511250 2024-09-10T22:00:37.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj63 testdata.511250 2024-09-10T22:00:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj64 testdata.511250 2024-09-10T22:00:37.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj65 testdata.511250 2024-09-10T22:00:37.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.753 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj66 testdata.511250 2024-09-10T22:00:37.886 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:37.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj67 testdata.511250 2024-09-10T22:00:38.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj68 testdata.511250 2024-09-10T22:00:38.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj69 testdata.511250 2024-09-10T22:00:38.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj70 testdata.511250 2024-09-10T22:00:38.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj71 testdata.511250 2024-09-10T22:00:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj72 testdata.511250 2024-09-10T22:00:38.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj73 testdata.511250 2024-09-10T22:00:38.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj74 testdata.511250 2024-09-10T22:00:38.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:38.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj75 testdata.511250 2024-09-10T22:00:39.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj76 testdata.511250 2024-09-10T22:00:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj77 testdata.511250 2024-09-10T22:00:39.478 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj78 testdata.511250 2024-09-10T22:00:39.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj79 testdata.511250 2024-09-10T22:00:39.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj80 testdata.511250 2024-09-10T22:00:39.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:39.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj81 testdata.511250 2024-09-10T22:00:40.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj82 testdata.511250 2024-09-10T22:00:40.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj83 testdata.511250 2024-09-10T22:00:40.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj84 testdata.511250 2024-09-10T22:00:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj85 testdata.511250 2024-09-10T22:00:40.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.611 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj86 testdata.511250 2024-09-10T22:00:40.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj87 testdata.511250 2024-09-10T22:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:40.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj88 testdata.511250 2024-09-10T22:00:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj89 testdata.511250 2024-09-10T22:00:41.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj90 testdata.511250 2024-09-10T22:00:41.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj91 testdata.511250 2024-09-10T22:00:41.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj92 testdata.511250 2024-09-10T22:00:41.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.672 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj93 testdata.511250 2024-09-10T22:00:41.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.814 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj94 testdata.511250 2024-09-10T22:00:41.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:41.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj95 testdata.511250 2024-09-10T22:00:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj96 testdata.511250 2024-09-10T22:00:42.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj97 testdata.511250 2024-09-10T22:00:42.411 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj98 testdata.511250 2024-09-10T22:00:42.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj99 testdata.511250 2024-09-10T22:00:42.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj100 testdata.511250 2024-09-10T22:00:42.822 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj101 testdata.511250 2024-09-10T22:00:42.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:42.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj102 testdata.511250 2024-09-10T22:00:43.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj103 testdata.511250 2024-09-10T22:00:43.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj104 testdata.511250 2024-09-10T22:00:43.421 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj105 testdata.511250 2024-09-10T22:00:43.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj106 testdata.511250 2024-09-10T22:00:43.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj107 testdata.511250 2024-09-10T22:00:43.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:43.897 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj108 testdata.511250 2024-09-10T22:00:44.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj109 testdata.511250 2024-09-10T22:00:44.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj110 testdata.511250 2024-09-10T22:00:44.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj111 testdata.511250 2024-09-10T22:00:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.458 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj112 testdata.511250 2024-09-10T22:00:44.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.617 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj113 testdata.511250 2024-09-10T22:00:44.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj114 testdata.511250 2024-09-10T22:00:44.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:44.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj115 testdata.511250 2024-09-10T22:00:45.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj116 testdata.511250 2024-09-10T22:00:45.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj117 testdata.511250 2024-09-10T22:00:45.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.305 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj118 testdata.511250 2024-09-10T22:00:45.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj119 testdata.511250 2024-09-10T22:00:45.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj120 testdata.511250 2024-09-10T22:00:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj121 testdata.511250 2024-09-10T22:00:45.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:45.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj122 testdata.511250 2024-09-10T22:00:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.052 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj123 testdata.511250 2024-09-10T22:00:46.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.195 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj124 testdata.511250 2024-09-10T22:00:46.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj125 testdata.511250 2024-09-10T22:00:46.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.504 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj126 testdata.511250 2024-09-10T22:00:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj127 testdata.511250 2024-09-10T22:00:46.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj128 testdata.511250 2024-09-10T22:00:46.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:46.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj129 testdata.511250 2024-09-10T22:00:47.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:47.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj130 testdata.511250 2024-09-10T22:00:47.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:47.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj131 testdata.511250 2024-09-10T22:00:47.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:47.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj132 testdata.511250 2024-09-10T22:00:47.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:47.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj133 testdata.511250 2024-09-10T22:00:47.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:47.954 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj134 testdata.511250 2024-09-10T22:00:48.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj135 testdata.511250 2024-09-10T22:00:48.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj136 testdata.511250 2024-09-10T22:00:48.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj137 testdata.511250 2024-09-10T22:00:48.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj138 testdata.511250 2024-09-10T22:00:48.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj139 testdata.511250 2024-09-10T22:00:48.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj140 testdata.511250 2024-09-10T22:00:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:48.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj141 testdata.511250 2024-09-10T22:00:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj142 testdata.511250 2024-09-10T22:00:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj143 testdata.511250 2024-09-10T22:00:49.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.423 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj144 testdata.511250 2024-09-10T22:00:49.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj145 testdata.511250 2024-09-10T22:00:49.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj146 testdata.511250 2024-09-10T22:00:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:49.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj147 testdata.511250 2024-09-10T22:00:50.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj148 testdata.511250 2024-09-10T22:00:50.193 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.194 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj149 testdata.511250 2024-09-10T22:00:50.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.336 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj150 testdata.511250 2024-09-10T22:00:50.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.470 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj151 testdata.511250 2024-09-10T22:00:50.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj152 testdata.511250 2024-09-10T22:00:50.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj153 testdata.511250 2024-09-10T22:00:50.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:50.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj154 testdata.511250 2024-09-10T22:00:51.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj155 testdata.511250 2024-09-10T22:00:51.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj156 testdata.511250 2024-09-10T22:00:51.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.308 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj157 testdata.511250 2024-09-10T22:00:51.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj158 testdata.511250 2024-09-10T22:00:51.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.629 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj159 testdata.511250 2024-09-10T22:00:51.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj160 testdata.511250 2024-09-10T22:00:51.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:51.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj161 testdata.511250 2024-09-10T22:00:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.048 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj162 testdata.511250 2024-09-10T22:00:52.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.190 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj163 testdata.511250 2024-09-10T22:00:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj164 testdata.511250 2024-09-10T22:00:52.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj165 testdata.511250 2024-09-10T22:00:52.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj166 testdata.511250 2024-09-10T22:00:52.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj167 testdata.511250 2024-09-10T22:00:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:52.894 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj168 testdata.511250 2024-09-10T22:00:53.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj169 testdata.511250 2024-09-10T22:00:53.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj170 testdata.511250 2024-09-10T22:00:53.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj171 testdata.511250 2024-09-10T22:00:53.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj172 testdata.511250 2024-09-10T22:00:53.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj173 testdata.511250 2024-09-10T22:00:53.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:53.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj174 testdata.511250 2024-09-10T22:00:54.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj175 testdata.511250 2024-09-10T22:00:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.227 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj176 testdata.511250 2024-09-10T22:00:54.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.377 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj177 testdata.511250 2024-09-10T22:00:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj178 testdata.511250 2024-09-10T22:00:54.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj179 testdata.511250 2024-09-10T22:00:54.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj180 testdata.511250 2024-09-10T22:00:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:54.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj181 testdata.511250 2024-09-10T22:00:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj182 testdata.511250 2024-09-10T22:00:55.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj183 testdata.511250 2024-09-10T22:00:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.357 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj184 testdata.511250 2024-09-10T22:00:55.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj185 testdata.511250 2024-09-10T22:00:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj186 testdata.511250 2024-09-10T22:00:55.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj187 testdata.511250 2024-09-10T22:00:55.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:55.961 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj188 testdata.511250 2024-09-10T22:00:56.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj189 testdata.511250 2024-09-10T22:00:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj190 testdata.511250 2024-09-10T22:00:56.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj191 testdata.511250 2024-09-10T22:00:56.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj192 testdata.511250 2024-09-10T22:00:56.647 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj193 testdata.511250 2024-09-10T22:00:56.789 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.790 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj194 testdata.511250 2024-09-10T22:00:56.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:56.970 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj195 testdata.511250 2024-09-10T22:00:57.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.116 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj196 testdata.511250 2024-09-10T22:00:57.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj197 testdata.511250 2024-09-10T22:00:57.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj198 testdata.511250 2024-09-10T22:00:57.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj199 testdata.511250 2024-09-10T22:00:57.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj200 testdata.511250 2024-09-10T22:00:57.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj201 testdata.511250 2024-09-10T22:00:57.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:57.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj202 testdata.511250 2024-09-10T22:00:58.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj203 testdata.511250 2024-09-10T22:00:58.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj204 testdata.511250 2024-09-10T22:00:58.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj205 testdata.511250 2024-09-10T22:00:58.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj206 testdata.511250 2024-09-10T22:00:58.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.684 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj207 testdata.511250 2024-09-10T22:00:58.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj208 testdata.511250 2024-09-10T22:00:58.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:58.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj209 testdata.511250 2024-09-10T22:00:59.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.094 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj210 testdata.511250 2024-09-10T22:00:59.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj211 testdata.511250 2024-09-10T22:00:59.379 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj212 testdata.511250 2024-09-10T22:00:59.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj213 testdata.511250 2024-09-10T22:00:59.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj214 testdata.511250 2024-09-10T22:00:59.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj215 testdata.511250 2024-09-10T22:00:59.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:00:59.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj216 testdata.511250 2024-09-10T22:01:00.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj217 testdata.511250 2024-09-10T22:01:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj218 testdata.511250 2024-09-10T22:01:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj219 testdata.511250 2024-09-10T22:01:00.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj220 testdata.511250 2024-09-10T22:01:00.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj221 testdata.511250 2024-09-10T22:01:00.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.828 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj222 testdata.511250 2024-09-10T22:01:00.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:00.962 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj223 testdata.511250 2024-09-10T22:01:01.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj224 testdata.511250 2024-09-10T22:01:01.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj225 testdata.511250 2024-09-10T22:01:01.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj226 testdata.511250 2024-09-10T22:01:01.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj227 testdata.511250 2024-09-10T22:01:01.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.658 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj228 testdata.511250 2024-09-10T22:01:01.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj229 testdata.511250 2024-09-10T22:01:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:01.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj230 testdata.511250 2024-09-10T22:01:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj231 testdata.511250 2024-09-10T22:01:02.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj232 testdata.511250 2024-09-10T22:01:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj233 testdata.511250 2024-09-10T22:01:02.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj234 testdata.511250 2024-09-10T22:01:02.696 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj235 testdata.511250 2024-09-10T22:01:02.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj236 testdata.511250 2024-09-10T22:01:02.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:02.981 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj237 testdata.511250 2024-09-10T22:01:03.123 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.124 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj238 testdata.511250 2024-09-10T22:01:03.257 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj239 testdata.511250 2024-09-10T22:01:03.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj240 testdata.511250 2024-09-10T22:01:03.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj241 testdata.511250 2024-09-10T22:01:03.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj242 testdata.511250 2024-09-10T22:01:03.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:03.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj243 testdata.511250 2024-09-10T22:01:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.137 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj244 testdata.511250 2024-09-10T22:01:04.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.271 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj245 testdata.511250 2024-09-10T22:01:04.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.405 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj246 testdata.511250 2024-09-10T22:01:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj247 testdata.511250 2024-09-10T22:01:04.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj248 testdata.511250 2024-09-10T22:01:04.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj249 testdata.511250 2024-09-10T22:01:04.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:04.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj250 testdata.511250 2024-09-10T22:01:05.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj251 testdata.511250 2024-09-10T22:01:05.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.251 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj252 testdata.511250 2024-09-10T22:01:05.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj253 testdata.511250 2024-09-10T22:01:05.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj254 testdata.511250 2024-09-10T22:01:05.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj255 testdata.511250 2024-09-10T22:01:05.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:05.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj256 testdata.511250 2024-09-10T22:01:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.030 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj257 testdata.511250 2024-09-10T22:01:06.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj258 testdata.511250 2024-09-10T22:01:06.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.315 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj259 testdata.511250 2024-09-10T22:01:06.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.449 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj260 testdata.511250 2024-09-10T22:01:06.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj261 testdata.511250 2024-09-10T22:01:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj262 testdata.511250 2024-09-10T22:01:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj263 testdata.511250 2024-09-10T22:01:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:06.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj264 testdata.511250 2024-09-10T22:01:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj265 testdata.511250 2024-09-10T22:01:07.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj266 testdata.511250 2024-09-10T22:01:07.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj267 testdata.511250 2024-09-10T22:01:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj268 testdata.511250 2024-09-10T22:01:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj269 testdata.511250 2024-09-10T22:01:07.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:07.923 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj270 testdata.511250 2024-09-10T22:01:08.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj271 testdata.511250 2024-09-10T22:01:08.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj272 testdata.511250 2024-09-10T22:01:08.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj273 testdata.511250 2024-09-10T22:01:08.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.493 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj274 testdata.511250 2024-09-10T22:01:08.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj275 testdata.511250 2024-09-10T22:01:08.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.794 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj276 testdata.511250 2024-09-10T22:01:08.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:08.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj277 testdata.511250 2024-09-10T22:01:09.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj278 testdata.511250 2024-09-10T22:01:09.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj279 testdata.511250 2024-09-10T22:01:09.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.355 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj280 testdata.511250 2024-09-10T22:01:09.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj281 testdata.511250 2024-09-10T22:01:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj282 testdata.511250 2024-09-10T22:01:09.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj283 testdata.511250 2024-09-10T22:01:09.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:09.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj284 testdata.511250 2024-09-10T22:01:10.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj285 testdata.511250 2024-09-10T22:01:10.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj286 testdata.511250 2024-09-10T22:01:10.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.335 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj287 testdata.511250 2024-09-10T22:01:10.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj288 testdata.511250 2024-09-10T22:01:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj289 testdata.511250 2024-09-10T22:01:10.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj290 testdata.511250 2024-09-10T22:01:10.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:10.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj291 testdata.511250 2024-09-10T22:01:11.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj292 testdata.511250 2024-09-10T22:01:11.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj293 testdata.511250 2024-09-10T22:01:11.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj294 testdata.511250 2024-09-10T22:01:11.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj295 testdata.511250 2024-09-10T22:01:11.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj296 testdata.511250 2024-09-10T22:01:11.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.809 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj297 testdata.511250 2024-09-10T22:01:11.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:11.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj298 testdata.511250 2024-09-10T22:01:12.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj299 testdata.511250 2024-09-10T22:01:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj300 testdata.511250 2024-09-10T22:01:12.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj301 testdata.511250 2024-09-10T22:01:12.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj302 testdata.511250 2024-09-10T22:01:12.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj303 testdata.511250 2024-09-10T22:01:12.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj304 testdata.511250 2024-09-10T22:01:12.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:12.949 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj305 testdata.511250 2024-09-10T22:01:13.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj306 testdata.511250 2024-09-10T22:01:13.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj307 testdata.511250 2024-09-10T22:01:13.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj308 testdata.511250 2024-09-10T22:01:13.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj309 testdata.511250 2024-09-10T22:01:13.702 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.703 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj310 testdata.511250 2024-09-10T22:01:13.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:13.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj311 testdata.511250 2024-09-10T22:01:14.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj312 testdata.511250 2024-09-10T22:01:14.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.146 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj313 testdata.511250 2024-09-10T22:01:14.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj314 testdata.511250 2024-09-10T22:01:14.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.422 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj315 testdata.511250 2024-09-10T22:01:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj316 testdata.511250 2024-09-10T22:01:14.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj317 testdata.511250 2024-09-10T22:01:14.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:14.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj318 testdata.511250 2024-09-10T22:01:15.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj319 testdata.511250 2024-09-10T22:01:15.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj320 testdata.511250 2024-09-10T22:01:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.277 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj321 testdata.511250 2024-09-10T22:01:15.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj322 testdata.511250 2024-09-10T22:01:15.620 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj323 testdata.511250 2024-09-10T22:01:15.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.755 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj324 testdata.511250 2024-09-10T22:01:15.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:15.888 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj325 testdata.511250 2024-09-10T22:01:16.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj326 testdata.511250 2024-09-10T22:01:16.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj327 testdata.511250 2024-09-10T22:01:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.307 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj328 testdata.511250 2024-09-10T22:01:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.441 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj329 testdata.511250 2024-09-10T22:01:16.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.575 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj330 testdata.511250 2024-09-10T22:01:16.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.718 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj331 testdata.511250 2024-09-10T22:01:16.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:16.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj332 testdata.511250 2024-09-10T22:01:17.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj333 testdata.511250 2024-09-10T22:01:17.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj334 testdata.511250 2024-09-10T22:01:17.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj335 testdata.511250 2024-09-10T22:01:17.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj336 testdata.511250 2024-09-10T22:01:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj337 testdata.511250 2024-09-10T22:01:17.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.749 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj338 testdata.511250 2024-09-10T22:01:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:17.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj339 testdata.511250 2024-09-10T22:01:18.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:18.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj340 testdata.511250 2024-09-10T22:01:18.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:18.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj341 testdata.511250 2024-09-10T22:01:18.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:18.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj342 testdata.511250 2024-09-10T22:01:18.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:18.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj343 testdata.511250 2024-09-10T22:01:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:18.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj344 testdata.511250 2024-09-10T22:01:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj345 testdata.511250 2024-09-10T22:01:19.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj346 testdata.511250 2024-09-10T22:01:19.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj347 testdata.511250 2024-09-10T22:01:19.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj348 testdata.511250 2024-09-10T22:01:19.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj349 testdata.511250 2024-09-10T22:01:19.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj350 testdata.511250 2024-09-10T22:01:19.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:19.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj351 testdata.511250 2024-09-10T22:01:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj352 testdata.511250 2024-09-10T22:01:20.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.260 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj353 testdata.511250 2024-09-10T22:01:20.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj354 testdata.511250 2024-09-10T22:01:20.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj355 testdata.511250 2024-09-10T22:01:20.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj356 testdata.511250 2024-09-10T22:01:20.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj357 testdata.511250 2024-09-10T22:01:20.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:20.938 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj358 testdata.511250 2024-09-10T22:01:21.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj359 testdata.511250 2024-09-10T22:01:21.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj360 testdata.511250 2024-09-10T22:01:21.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj361 testdata.511250 2024-09-10T22:01:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj362 testdata.511250 2024-09-10T22:01:21.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj363 testdata.511250 2024-09-10T22:01:21.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj364 testdata.511250 2024-09-10T22:01:21.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:21.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj365 testdata.511250 2024-09-10T22:01:22.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.103 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj366 testdata.511250 2024-09-10T22:01:22.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj367 testdata.511250 2024-09-10T22:01:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj368 testdata.511250 2024-09-10T22:01:22.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj369 testdata.511250 2024-09-10T22:01:22.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj370 testdata.511250 2024-09-10T22:01:22.797 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj371 testdata.511250 2024-09-10T22:01:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:22.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj372 testdata.511250 2024-09-10T22:01:23.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj373 testdata.511250 2024-09-10T22:01:23.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.241 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj374 testdata.511250 2024-09-10T22:01:23.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj375 testdata.511250 2024-09-10T22:01:23.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.543 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj376 testdata.511250 2024-09-10T22:01:23.685 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.686 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj377 testdata.511250 2024-09-10T22:01:23.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj378 testdata.511250 2024-09-10T22:01:23.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:23.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj379 testdata.511250 2024-09-10T22:01:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj380 testdata.511250 2024-09-10T22:01:24.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj381 testdata.511250 2024-09-10T22:01:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj382 testdata.511250 2024-09-10T22:01:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.598 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj383 testdata.511250 2024-09-10T22:01:24.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj384 testdata.511250 2024-09-10T22:01:24.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:24.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj385 testdata.511250 2024-09-10T22:01:25.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.009 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj386 testdata.511250 2024-09-10T22:01:25.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.168 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj387 testdata.511250 2024-09-10T22:01:25.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.319 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj388 testdata.511250 2024-09-10T22:01:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj389 testdata.511250 2024-09-10T22:01:25.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj390 testdata.511250 2024-09-10T22:01:25.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj391 testdata.511250 2024-09-10T22:01:25.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:25.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj392 testdata.511250 2024-09-10T22:01:26.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj393 testdata.511250 2024-09-10T22:01:26.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj394 testdata.511250 2024-09-10T22:01:26.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj395 testdata.511250 2024-09-10T22:01:26.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj396 testdata.511250 2024-09-10T22:01:26.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj397 testdata.511250 2024-09-10T22:01:26.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.776 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj398 testdata.511250 2024-09-10T22:01:26.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:26.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj399 testdata.511250 2024-09-10T22:01:27.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj400 testdata.511250 2024-09-10T22:01:27.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj401 testdata.511250 2024-09-10T22:01:27.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.329 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj402 testdata.511250 2024-09-10T22:01:27.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.480 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj403 testdata.511250 2024-09-10T22:01:27.639 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj404 testdata.511250 2024-09-10T22:01:27.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj405 testdata.511250 2024-09-10T22:01:27.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:27.916 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj406 testdata.511250 2024-09-10T22:01:28.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj407 testdata.511250 2024-09-10T22:01:28.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj408 testdata.511250 2024-09-10T22:01:28.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj409 testdata.511250 2024-09-10T22:01:28.468 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj410 testdata.511250 2024-09-10T22:01:28.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj411 testdata.511250 2024-09-10T22:01:28.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.762 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj412 testdata.511250 2024-09-10T22:01:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:28.896 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj413 testdata.511250 2024-09-10T22:01:29.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj414 testdata.511250 2024-09-10T22:01:29.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.181 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj415 testdata.511250 2024-09-10T22:01:29.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj416 testdata.511250 2024-09-10T22:01:29.490 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj417 testdata.511250 2024-09-10T22:01:29.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj418 testdata.511250 2024-09-10T22:01:29.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj419 testdata.511250 2024-09-10T22:01:29.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:29.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj420 testdata.511250 2024-09-10T22:01:30.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj421 testdata.511250 2024-09-10T22:01:30.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj422 testdata.511250 2024-09-10T22:01:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj423 testdata.511250 2024-09-10T22:01:30.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj424 testdata.511250 2024-09-10T22:01:30.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj425 testdata.511250 2024-09-10T22:01:30.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj426 testdata.511250 2024-09-10T22:01:30.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:30.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj427 testdata.511250 2024-09-10T22:01:31.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.015 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj428 testdata.511250 2024-09-10T22:01:31.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.149 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj429 testdata.511250 2024-09-10T22:01:31.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj430 testdata.511250 2024-09-10T22:01:31.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.442 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj431 testdata.511250 2024-09-10T22:01:31.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.610 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj432 testdata.511250 2024-09-10T22:01:31.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj433 testdata.511250 2024-09-10T22:01:31.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:31.903 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj434 testdata.511250 2024-09-10T22:01:32.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj435 testdata.511250 2024-09-10T22:01:32.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj436 testdata.511250 2024-09-10T22:01:32.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj437 testdata.511250 2024-09-10T22:01:32.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.498 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj438 testdata.511250 2024-09-10T22:01:32.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.640 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj439 testdata.511250 2024-09-10T22:01:32.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.775 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj440 testdata.511250 2024-09-10T22:01:32.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:32.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj441 testdata.511250 2024-09-10T22:01:33.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj442 testdata.511250 2024-09-10T22:01:33.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj443 testdata.511250 2024-09-10T22:01:33.343 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj444 testdata.511250 2024-09-10T22:01:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj445 testdata.511250 2024-09-10T22:01:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.654 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj446 testdata.511250 2024-09-10T22:01:33.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj447 testdata.511250 2024-09-10T22:01:33.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:33.922 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj448 testdata.511250 2024-09-10T22:01:34.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj449 testdata.511250 2024-09-10T22:01:34.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj450 testdata.511250 2024-09-10T22:01:34.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj451 testdata.511250 2024-09-10T22:01:34.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj452 testdata.511250 2024-09-10T22:01:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.659 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj453 testdata.511250 2024-09-10T22:01:34.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.801 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj454 testdata.511250 2024-09-10T22:01:34.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:34.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj455 testdata.511250 2024-09-10T22:01:35.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.087 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj456 testdata.511250 2024-09-10T22:01:35.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.220 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj457 testdata.511250 2024-09-10T22:01:35.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.371 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj458 testdata.511250 2024-09-10T22:01:35.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj459 testdata.511250 2024-09-10T22:01:35.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj460 testdata.511250 2024-09-10T22:01:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj461 testdata.511250 2024-09-10T22:01:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:35.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj462 testdata.511250 2024-09-10T22:01:36.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.100 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj463 testdata.511250 2024-09-10T22:01:36.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj464 testdata.511250 2024-09-10T22:01:36.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.376 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj465 testdata.511250 2024-09-10T22:01:36.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj466 testdata.511250 2024-09-10T22:01:36.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.653 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj467 testdata.511250 2024-09-10T22:01:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj468 testdata.511250 2024-09-10T22:01:36.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:36.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj469 testdata.511250 2024-09-10T22:01:37.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj470 testdata.511250 2024-09-10T22:01:37.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj471 testdata.511250 2024-09-10T22:01:37.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.373 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj472 testdata.511250 2024-09-10T22:01:37.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj473 testdata.511250 2024-09-10T22:01:37.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj474 testdata.511250 2024-09-10T22:01:37.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj475 testdata.511250 2024-09-10T22:01:37.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:37.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj476 testdata.511250 2024-09-10T22:01:38.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj477 testdata.511250 2024-09-10T22:01:38.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj478 testdata.511250 2024-09-10T22:01:38.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj479 testdata.511250 2024-09-10T22:01:38.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj480 testdata.511250 2024-09-10T22:01:38.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj481 testdata.511250 2024-09-10T22:01:38.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj482 testdata.511250 2024-09-10T22:01:38.964 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:38.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj483 testdata.511250 2024-09-10T22:01:39.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.098 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj484 testdata.511250 2024-09-10T22:01:39.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj485 testdata.511250 2024-09-10T22:01:39.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.433 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj486 testdata.511250 2024-09-10T22:01:39.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj487 testdata.511250 2024-09-10T22:01:39.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj488 testdata.511250 2024-09-10T22:01:39.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:39.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj489 testdata.511250 2024-09-10T22:01:40.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj490 testdata.511250 2024-09-10T22:01:40.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj491 testdata.511250 2024-09-10T22:01:40.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.310 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj492 testdata.511250 2024-09-10T22:01:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj493 testdata.511250 2024-09-10T22:01:40.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj494 testdata.511250 2024-09-10T22:01:40.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj495 testdata.511250 2024-09-10T22:01:40.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:40.877 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj496 testdata.511250 2024-09-10T22:01:41.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj497 testdata.511250 2024-09-10T22:01:41.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj498 testdata.511250 2024-09-10T22:01:41.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj499 testdata.511250 2024-09-10T22:01:41.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj500 testdata.511250 2024-09-10T22:01:41.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj501 testdata.511250 2024-09-10T22:01:41.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj502 testdata.511250 2024-09-10T22:01:41.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:41.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj503 testdata.511250 2024-09-10T22:01:42.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj504 testdata.511250 2024-09-10T22:01:42.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj505 testdata.511250 2024-09-10T22:01:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj506 testdata.511250 2024-09-10T22:01:42.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj507 testdata.511250 2024-09-10T22:01:42.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj508 testdata.511250 2024-09-10T22:01:42.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj509 testdata.511250 2024-09-10T22:01:42.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:42.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj510 testdata.511250 2024-09-10T22:01:43.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.077 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj511 testdata.511250 2024-09-10T22:01:43.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj512 testdata.511250 2024-09-10T22:01:43.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj513 testdata.511250 2024-09-10T22:01:43.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj514 testdata.511250 2024-09-10T22:01:43.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj515 testdata.511250 2024-09-10T22:01:43.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj516 testdata.511250 2024-09-10T22:01:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:43.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj517 testdata.511250 2024-09-10T22:01:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj518 testdata.511250 2024-09-10T22:01:44.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj519 testdata.511250 2024-09-10T22:01:44.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.363 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj520 testdata.511250 2024-09-10T22:01:44.505 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj521 testdata.511250 2024-09-10T22:01:44.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.648 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj522 testdata.511250 2024-09-10T22:01:44.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj523 testdata.511250 2024-09-10T22:01:44.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:44.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj524 testdata.511250 2024-09-10T22:01:45.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj525 testdata.511250 2024-09-10T22:01:45.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj526 testdata.511250 2024-09-10T22:01:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj527 testdata.511250 2024-09-10T22:01:45.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj528 testdata.511250 2024-09-10T22:01:45.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.770 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj529 testdata.511250 2024-09-10T22:01:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:45.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj530 testdata.511250 2024-09-10T22:01:46.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.038 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj531 testdata.511250 2024-09-10T22:01:46.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.189 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj532 testdata.511250 2024-09-10T22:01:46.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.340 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj533 testdata.511250 2024-09-10T22:01:46.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.474 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj534 testdata.511250 2024-09-10T22:01:46.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj535 testdata.511250 2024-09-10T22:01:46.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.750 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj536 testdata.511250 2024-09-10T22:01:46.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:46.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj537 testdata.511250 2024-09-10T22:01:47.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj538 testdata.511250 2024-09-10T22:01:47.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj539 testdata.511250 2024-09-10T22:01:47.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.320 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj540 testdata.511250 2024-09-10T22:01:47.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.479 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj541 testdata.511250 2024-09-10T22:01:47.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj542 testdata.511250 2024-09-10T22:01:47.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj543 testdata.511250 2024-09-10T22:01:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:47.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj544 testdata.511250 2024-09-10T22:01:48.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj545 testdata.511250 2024-09-10T22:01:48.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj546 testdata.511250 2024-09-10T22:01:48.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj547 testdata.511250 2024-09-10T22:01:48.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj548 testdata.511250 2024-09-10T22:01:48.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj549 testdata.511250 2024-09-10T22:01:48.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj550 testdata.511250 2024-09-10T22:01:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:48.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj551 testdata.511250 2024-09-10T22:01:49.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.046 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj552 testdata.511250 2024-09-10T22:01:49.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj553 testdata.511250 2024-09-10T22:01:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.322 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj554 testdata.511250 2024-09-10T22:01:49.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj555 testdata.511250 2024-09-10T22:01:49.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj556 testdata.511250 2024-09-10T22:01:49.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj557 testdata.511250 2024-09-10T22:01:49.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:49.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj558 testdata.511250 2024-09-10T22:01:50.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.276 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj559 testdata.511250 2024-09-10T22:01:50.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj560 testdata.511250 2024-09-10T22:01:50.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj561 testdata.511250 2024-09-10T22:01:50.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj562 testdata.511250 2024-09-10T22:01:50.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.813 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj563 testdata.511250 2024-09-10T22:01:50.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:50.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj564 testdata.511250 2024-09-10T22:01:51.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.089 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj565 testdata.511250 2024-09-10T22:01:51.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj566 testdata.511250 2024-09-10T22:01:51.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj567 testdata.511250 2024-09-10T22:01:51.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj568 testdata.511250 2024-09-10T22:01:51.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj569 testdata.511250 2024-09-10T22:01:51.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj570 testdata.511250 2024-09-10T22:01:51.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:51.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj571 testdata.511250 2024-09-10T22:01:52.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.045 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj572 testdata.511250 2024-09-10T22:01:52.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj573 testdata.511250 2024-09-10T22:01:52.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj574 testdata.511250 2024-09-10T22:01:52.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj575 testdata.511250 2024-09-10T22:01:52.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj576 testdata.511250 2024-09-10T22:01:52.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.723 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj577 testdata.511250 2024-09-10T22:01:52.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.849 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj578 testdata.511250 2024-09-10T22:01:52.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:52.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj579 testdata.511250 2024-09-10T22:01:53.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.108 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj580 testdata.511250 2024-09-10T22:01:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj581 testdata.511250 2024-09-10T22:01:53.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.401 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj582 testdata.511250 2024-09-10T22:01:53.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.553 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj583 testdata.511250 2024-09-10T22:01:53.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj584 testdata.511250 2024-09-10T22:01:53.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj585 testdata.511250 2024-09-10T22:01:53.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:53.963 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj586 testdata.511250 2024-09-10T22:01:54.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj587 testdata.511250 2024-09-10T22:01:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.256 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj588 testdata.511250 2024-09-10T22:01:54.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj589 testdata.511250 2024-09-10T22:01:54.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj590 testdata.511250 2024-09-10T22:01:54.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj591 testdata.511250 2024-09-10T22:01:54.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.759 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj592 testdata.511250 2024-09-10T22:01:54.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:54.885 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj593 testdata.511250 2024-09-10T22:01:55.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.010 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj594 testdata.511250 2024-09-10T22:01:55.152 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj595 testdata.511250 2024-09-10T22:01:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj596 testdata.511250 2024-09-10T22:01:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj597 testdata.511250 2024-09-10T22:01:55.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.689 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj598 testdata.511250 2024-09-10T22:01:55.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.823 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj599 testdata.511250 2024-09-10T22:01:55.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:55.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj600 testdata.511250 2024-09-10T22:01:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj601 testdata.511250 2024-09-10T22:01:56.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj602 testdata.511250 2024-09-10T22:01:56.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj603 testdata.511250 2024-09-10T22:01:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.476 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj604 testdata.511250 2024-09-10T22:01:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj605 testdata.511250 2024-09-10T22:01:56.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj606 testdata.511250 2024-09-10T22:01:56.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:56.895 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj607 testdata.511250 2024-09-10T22:01:57.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj608 testdata.511250 2024-09-10T22:01:57.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj609 testdata.511250 2024-09-10T22:01:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj610 testdata.511250 2024-09-10T22:01:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.428 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj611 testdata.511250 2024-09-10T22:01:57.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.599 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj612 testdata.511250 2024-09-10T22:01:57.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj613 testdata.511250 2024-09-10T22:01:57.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj614 testdata.511250 2024-09-10T22:01:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:57.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj615 testdata.511250 2024-09-10T22:01:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj616 testdata.511250 2024-09-10T22:01:58.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj617 testdata.511250 2024-09-10T22:01:58.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj618 testdata.511250 2024-09-10T22:01:58.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.512 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj619 testdata.511250 2024-09-10T22:01:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj620 testdata.511250 2024-09-10T22:01:58.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj621 testdata.511250 2024-09-10T22:01:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:58.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj622 testdata.511250 2024-09-10T22:01:59.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj623 testdata.511250 2024-09-10T22:01:59.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj624 testdata.511250 2024-09-10T22:01:59.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj625 testdata.511250 2024-09-10T22:01:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.436 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj626 testdata.511250 2024-09-10T22:01:59.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj627 testdata.511250 2024-09-10T22:01:59.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.736 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj628 testdata.511250 2024-09-10T22:01:59.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:01:59.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj629 testdata.511250 2024-09-10T22:02:00.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.004 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj630 testdata.511250 2024-09-10T22:02:00.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj631 testdata.511250 2024-09-10T22:02:00.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj632 testdata.511250 2024-09-10T22:02:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj633 testdata.511250 2024-09-10T22:02:00.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj634 testdata.511250 2024-09-10T22:02:00.682 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj635 testdata.511250 2024-09-10T22:02:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj636 testdata.511250 2024-09-10T22:02:00.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:00.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj637 testdata.511250 2024-09-10T22:02:01.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.068 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj638 testdata.511250 2024-09-10T22:02:01.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj639 testdata.511250 2024-09-10T22:02:01.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj640 testdata.511250 2024-09-10T22:02:01.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.513 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj641 testdata.511250 2024-09-10T22:02:01.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj642 testdata.511250 2024-09-10T22:02:01.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.780 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj643 testdata.511250 2024-09-10T22:02:01.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:01.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj644 testdata.511250 2024-09-10T22:02:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj645 testdata.511250 2024-09-10T22:02:02.165 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.166 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj646 testdata.511250 2024-09-10T22:02:02.316 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.317 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj647 testdata.511250 2024-09-10T22:02:02.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.450 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj648 testdata.511250 2024-09-10T22:02:02.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj649 testdata.511250 2024-09-10T22:02:02.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.710 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj650 testdata.511250 2024-09-10T22:02:02.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj651 testdata.511250 2024-09-10T22:02:02.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:02.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj652 testdata.511250 2024-09-10T22:02:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj653 testdata.511250 2024-09-10T22:02:03.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.238 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj654 testdata.511250 2024-09-10T22:02:03.380 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj655 testdata.511250 2024-09-10T22:02:03.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.523 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj656 testdata.511250 2024-09-10T22:02:03.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj657 testdata.511250 2024-09-10T22:02:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj658 testdata.511250 2024-09-10T22:02:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:03.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj659 testdata.511250 2024-09-10T22:02:04.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.067 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj660 testdata.511250 2024-09-10T22:02:04.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.202 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj661 testdata.511250 2024-09-10T22:02:04.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.327 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj662 testdata.511250 2024-09-10T22:02:04.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.462 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj663 testdata.511250 2024-09-10T22:02:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.587 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj664 testdata.511250 2024-09-10T22:02:04.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.721 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj665 testdata.511250 2024-09-10T22:02:04.863 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.864 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj666 testdata.511250 2024-09-10T22:02:04.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:04.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj667 testdata.511250 2024-09-10T22:02:05.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj668 testdata.511250 2024-09-10T22:02:05.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj669 testdata.511250 2024-09-10T22:02:05.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj670 testdata.511250 2024-09-10T22:02:05.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.609 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj671 testdata.511250 2024-09-10T22:02:05.751 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.752 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj672 testdata.511250 2024-09-10T22:02:05.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:05.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj673 testdata.511250 2024-09-10T22:02:06.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.078 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj674 testdata.511250 2024-09-10T22:02:06.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj675 testdata.511250 2024-09-10T22:02:06.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj676 testdata.511250 2024-09-10T22:02:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.472 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj677 testdata.511250 2024-09-10T22:02:06.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj678 testdata.511250 2024-09-10T22:02:06.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj679 testdata.511250 2024-09-10T22:02:06.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj680 testdata.511250 2024-09-10T22:02:06.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:06.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj681 testdata.511250 2024-09-10T22:02:07.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj682 testdata.511250 2024-09-10T22:02:07.242 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj683 testdata.511250 2024-09-10T22:02:07.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.427 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj684 testdata.511250 2024-09-10T22:02:07.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj685 testdata.511250 2024-09-10T22:02:07.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.704 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj686 testdata.511250 2024-09-10T22:02:07.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj687 testdata.511250 2024-09-10T22:02:07.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:07.980 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj688 testdata.511250 2024-09-10T22:02:08.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.122 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj689 testdata.511250 2024-09-10T22:02:08.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj690 testdata.511250 2024-09-10T22:02:08.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj691 testdata.511250 2024-09-10T22:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj692 testdata.511250 2024-09-10T22:02:08.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj693 testdata.511250 2024-09-10T22:02:08.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj694 testdata.511250 2024-09-10T22:02:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:08.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj695 testdata.511250 2024-09-10T22:02:09.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj696 testdata.511250 2024-09-10T22:02:09.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.155 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj697 testdata.511250 2024-09-10T22:02:09.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj698 testdata.511250 2024-09-10T22:02:09.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj699 testdata.511250 2024-09-10T22:02:09.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj700 testdata.511250 2024-09-10T22:02:09.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj701 testdata.511250 2024-09-10T22:02:09.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.842 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj702 testdata.511250 2024-09-10T22:02:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:09.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj703 testdata.511250 2024-09-10T22:02:10.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj704 testdata.511250 2024-09-10T22:02:10.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj705 testdata.511250 2024-09-10T22:02:10.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.362 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj706 testdata.511250 2024-09-10T22:02:10.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj707 testdata.511250 2024-09-10T22:02:10.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj708 testdata.511250 2024-09-10T22:02:10.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj709 testdata.511250 2024-09-10T22:02:10.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:10.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj710 testdata.511250 2024-09-10T22:02:11.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.041 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj711 testdata.511250 2024-09-10T22:02:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj712 testdata.511250 2024-09-10T22:02:11.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj713 testdata.511250 2024-09-10T22:02:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.451 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj714 testdata.511250 2024-09-10T22:02:11.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj715 testdata.511250 2024-09-10T22:02:11.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.745 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj716 testdata.511250 2024-09-10T22:02:11.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:11.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj717 testdata.511250 2024-09-10T22:02:12.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.005 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj718 testdata.511250 2024-09-10T22:02:12.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.147 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj719 testdata.511250 2024-09-10T22:02:12.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj720 testdata.511250 2024-09-10T22:02:12.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.407 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj721 testdata.511250 2024-09-10T22:02:12.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj722 testdata.511250 2024-09-10T22:02:12.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj723 testdata.511250 2024-09-10T22:02:12.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj724 testdata.511250 2024-09-10T22:02:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:12.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj725 testdata.511250 2024-09-10T22:02:13.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.044 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj726 testdata.511250 2024-09-10T22:02:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj727 testdata.511250 2024-09-10T22:02:13.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj728 testdata.511250 2024-09-10T22:02:13.437 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.438 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj729 testdata.511250 2024-09-10T22:02:13.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj730 testdata.511250 2024-09-10T22:02:13.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj731 testdata.511250 2024-09-10T22:02:13.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:13.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj732 testdata.511250 2024-09-10T22:02:14.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj733 testdata.511250 2024-09-10T22:02:14.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj734 testdata.511250 2024-09-10T22:02:14.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj735 testdata.511250 2024-09-10T22:02:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.435 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj736 testdata.511250 2024-09-10T22:02:14.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.561 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj737 testdata.511250 2024-09-10T22:02:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj738 testdata.511250 2024-09-10T22:02:14.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj739 testdata.511250 2024-09-10T22:02:14.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:14.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj740 testdata.511250 2024-09-10T22:02:15.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj741 testdata.511250 2024-09-10T22:02:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj742 testdata.511250 2024-09-10T22:02:15.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.374 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj743 testdata.511250 2024-09-10T22:02:15.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj744 testdata.511250 2024-09-10T22:02:15.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj745 testdata.511250 2024-09-10T22:02:15.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj746 testdata.511250 2024-09-10T22:02:15.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:15.902 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj747 testdata.511250 2024-09-10T22:02:16.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj748 testdata.511250 2024-09-10T22:02:16.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj749 testdata.511250 2024-09-10T22:02:16.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj750 testdata.511250 2024-09-10T22:02:16.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj751 testdata.511250 2024-09-10T22:02:16.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj752 testdata.511250 2024-09-10T22:02:16.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.681 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj753 testdata.511250 2024-09-10T22:02:16.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj754 testdata.511250 2024-09-10T22:02:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:16.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj755 testdata.511250 2024-09-10T22:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.058 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj756 testdata.511250 2024-09-10T22:02:17.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.192 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj757 testdata.511250 2024-09-10T22:02:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.326 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj758 testdata.511250 2024-09-10T22:02:17.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.461 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj759 testdata.511250 2024-09-10T22:02:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj760 testdata.511250 2024-09-10T22:02:17.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj761 testdata.511250 2024-09-10T22:02:17.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:17.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj762 testdata.511250 2024-09-10T22:02:18.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj763 testdata.511250 2024-09-10T22:02:18.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.157 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj764 testdata.511250 2024-09-10T22:02:18.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.282 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj765 testdata.511250 2024-09-10T22:02:18.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj766 testdata.511250 2024-09-10T22:02:18.541 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.542 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj767 testdata.511250 2024-09-10T22:02:18.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj768 testdata.511250 2024-09-10T22:02:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.810 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj769 testdata.511250 2024-09-10T22:02:18.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:18.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj770 testdata.511250 2024-09-10T22:02:19.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj771 testdata.511250 2024-09-10T22:02:19.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.179 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj772 testdata.511250 2024-09-10T22:02:19.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.313 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj773 testdata.511250 2024-09-10T22:02:19.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj774 testdata.511250 2024-09-10T22:02:19.606 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj775 testdata.511250 2024-09-10T22:02:19.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj776 testdata.511250 2024-09-10T22:02:19.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj777 testdata.511250 2024-09-10T22:02:19.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:19.984 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj778 testdata.511250 2024-09-10T22:02:20.109 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj779 testdata.511250 2024-09-10T22:02:20.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj780 testdata.511250 2024-09-10T22:02:20.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.378 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj781 testdata.511250 2024-09-10T22:02:20.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj782 testdata.511250 2024-09-10T22:02:20.645 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj783 testdata.511250 2024-09-10T22:02:20.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.771 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj784 testdata.511250 2024-09-10T22:02:20.905 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:20.906 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj785 testdata.511250 2024-09-10T22:02:21.039 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:21.040 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj786 testdata.511250 2024-09-10T22:02:21.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:21.372 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj787 testdata.511250 2024-09-10T22:02:21.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:21.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj788 testdata.511250 2024-09-10T22:02:21.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:21.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj789 testdata.511250 2024-09-10T22:02:21.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:21.985 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj790 testdata.511250 2024-09-10T22:02:22.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj791 testdata.511250 2024-09-10T22:02:22.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj792 testdata.511250 2024-09-10T22:02:22.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.361 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj793 testdata.511250 2024-09-10T22:02:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.487 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj794 testdata.511250 2024-09-10T22:02:22.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj795 testdata.511250 2024-09-10T22:02:22.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.756 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj796 testdata.511250 2024-09-10T22:02:22.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:22.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj797 testdata.511250 2024-09-10T22:02:23.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.007 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj798 testdata.511250 2024-09-10T22:02:23.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.141 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj799 testdata.511250 2024-09-10T22:02:23.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.267 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj800 testdata.511250 2024-09-10T22:02:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.434 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj801 testdata.511250 2024-09-10T22:02:23.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj802 testdata.511250 2024-09-10T22:02:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.728 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj803 testdata.511250 2024-09-10T22:02:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj804 testdata.511250 2024-09-10T22:02:23.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:23.979 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj805 testdata.511250 2024-09-10T22:02:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj806 testdata.511250 2024-09-10T22:02:24.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj807 testdata.511250 2024-09-10T22:02:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.365 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj808 testdata.511250 2024-09-10T22:02:24.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.491 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj809 testdata.511250 2024-09-10T22:02:24.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj810 testdata.511250 2024-09-10T22:02:24.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj811 testdata.511250 2024-09-10T22:02:24.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:24.876 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj812 testdata.511250 2024-09-10T22:02:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.043 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj813 testdata.511250 2024-09-10T22:02:25.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.177 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj814 testdata.511250 2024-09-10T22:02:25.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.311 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj815 testdata.511250 2024-09-10T22:02:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj816 testdata.511250 2024-09-10T22:02:25.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj817 testdata.511250 2024-09-10T22:02:25.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj818 testdata.511250 2024-09-10T22:02:25.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:25.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj819 testdata.511250 2024-09-10T22:02:26.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.016 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj820 testdata.511250 2024-09-10T22:02:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj821 testdata.511250 2024-09-10T22:02:26.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj822 testdata.511250 2024-09-10T22:02:26.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj823 testdata.511250 2024-09-10T22:02:26.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj824 testdata.511250 2024-09-10T22:02:26.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj825 testdata.511250 2024-09-10T22:02:26.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:26.901 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj826 testdata.511250 2024-09-10T22:02:27.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.029 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj827 testdata.511250 2024-09-10T22:02:27.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj828 testdata.511250 2024-09-10T22:02:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj829 testdata.511250 2024-09-10T22:02:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj830 testdata.511250 2024-09-10T22:02:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.616 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj831 testdata.511250 2024-09-10T22:02:27.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj832 testdata.511250 2024-09-10T22:02:27.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:27.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj833 testdata.511250 2024-09-10T22:02:28.001 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj834 testdata.511250 2024-09-10T22:02:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj835 testdata.511250 2024-09-10T22:02:28.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.278 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj836 testdata.511250 2024-09-10T22:02:28.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj837 testdata.511250 2024-09-10T22:02:28.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj838 testdata.511250 2024-09-10T22:02:28.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj839 testdata.511250 2024-09-10T22:02:28.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj840 testdata.511250 2024-09-10T22:02:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:28.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj841 testdata.511250 2024-09-10T22:02:29.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj842 testdata.511250 2024-09-10T22:02:29.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.216 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj843 testdata.511250 2024-09-10T22:02:29.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj844 testdata.511250 2024-09-10T22:02:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj845 testdata.511250 2024-09-10T22:02:29.660 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj846 testdata.511250 2024-09-10T22:02:29.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj847 testdata.511250 2024-09-10T22:02:29.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:29.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj848 testdata.511250 2024-09-10T22:02:30.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj849 testdata.511250 2024-09-10T22:02:30.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.206 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj850 testdata.511250 2024-09-10T22:02:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj851 testdata.511250 2024-09-10T22:02:30.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj852 testdata.511250 2024-09-10T22:02:30.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj853 testdata.511250 2024-09-10T22:02:30.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj854 testdata.511250 2024-09-10T22:02:30.850 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.851 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj855 testdata.511250 2024-09-10T22:02:30.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:30.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj856 testdata.511250 2024-09-10T22:02:31.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj857 testdata.511250 2024-09-10T22:02:31.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj858 testdata.511250 2024-09-10T22:02:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj859 testdata.511250 2024-09-10T22:02:31.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj860 testdata.511250 2024-09-10T22:02:31.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj861 testdata.511250 2024-09-10T22:02:31.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj862 testdata.511250 2024-09-10T22:02:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:31.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj863 testdata.511250 2024-09-10T22:02:32.090 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.091 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj864 testdata.511250 2024-09-10T22:02:32.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj865 testdata.511250 2024-09-10T22:02:32.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj866 testdata.511250 2024-09-10T22:02:32.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj867 testdata.511250 2024-09-10T22:02:32.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.627 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj868 testdata.511250 2024-09-10T22:02:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj869 testdata.511250 2024-09-10T22:02:32.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:32.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj870 testdata.511250 2024-09-10T22:02:33.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.013 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj871 testdata.511250 2024-09-10T22:02:33.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj872 testdata.511250 2024-09-10T22:02:33.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.314 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj873 testdata.511250 2024-09-10T22:02:33.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.465 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj874 testdata.511250 2024-09-10T22:02:33.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.633 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj875 testdata.511250 2024-09-10T22:02:33.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.758 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj876 testdata.511250 2024-09-10T22:02:33.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:33.884 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj877 testdata.511250 2024-09-10T22:02:34.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj878 testdata.511250 2024-09-10T22:02:34.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj879 testdata.511250 2024-09-10T22:02:34.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj880 testdata.511250 2024-09-10T22:02:34.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj881 testdata.511250 2024-09-10T22:02:34.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj882 testdata.511250 2024-09-10T22:02:34.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.655 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj883 testdata.511250 2024-09-10T22:02:34.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj884 testdata.511250 2024-09-10T22:02:34.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:34.940 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj885 testdata.511250 2024-09-10T22:02:35.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.066 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj886 testdata.511250 2024-09-10T22:02:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj887 testdata.511250 2024-09-10T22:02:35.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.342 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj888 testdata.511250 2024-09-10T22:02:35.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj889 testdata.511250 2024-09-10T22:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.678 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj890 testdata.511250 2024-09-10T22:02:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj891 testdata.511250 2024-09-10T22:02:35.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:35.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj892 testdata.511250 2024-09-10T22:02:36.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj893 testdata.511250 2024-09-10T22:02:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj894 testdata.511250 2024-09-10T22:02:36.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj895 testdata.511250 2024-09-10T22:02:36.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.457 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj896 testdata.511250 2024-09-10T22:02:36.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.583 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj897 testdata.511250 2024-09-10T22:02:36.708 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.709 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj898 testdata.511250 2024-09-10T22:02:36.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj899 testdata.511250 2024-09-10T22:02:36.959 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:36.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj900 testdata.511250 2024-09-10T22:02:37.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.169 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj901 testdata.511250 2024-09-10T22:02:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.306 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj902 testdata.511250 2024-09-10T22:02:37.445 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj903 testdata.511250 2024-09-10T22:02:37.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.630 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj904 testdata.511250 2024-09-10T22:02:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.773 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj905 testdata.511250 2024-09-10T22:02:37.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:37.898 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj906 testdata.511250 2024-09-10T22:02:38.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj907 testdata.511250 2024-09-10T22:02:38.170 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj908 testdata.511250 2024-09-10T22:02:38.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.309 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj909 testdata.511250 2024-09-10T22:02:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.443 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj910 testdata.511250 2024-09-10T22:02:38.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj911 testdata.511250 2024-09-10T22:02:38.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj912 testdata.511250 2024-09-10T22:02:38.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj913 testdata.511250 2024-09-10T22:02:38.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:38.971 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj914 testdata.511250 2024-09-10T22:02:39.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.138 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj915 testdata.511250 2024-09-10T22:02:39.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.272 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj916 testdata.511250 2024-09-10T22:02:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj917 testdata.511250 2024-09-10T22:02:39.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj918 testdata.511250 2024-09-10T22:02:39.741 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj919 testdata.511250 2024-09-10T22:02:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.867 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj920 testdata.511250 2024-09-10T22:02:39.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:39.993 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj921 testdata.511250 2024-09-10T22:02:40.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj922 testdata.511250 2024-09-10T22:02:40.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj923 testdata.511250 2024-09-10T22:02:40.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.395 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj924 testdata.511250 2024-09-10T22:02:40.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj925 testdata.511250 2024-09-10T22:02:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj926 testdata.511250 2024-09-10T22:02:40.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj927 testdata.511250 2024-09-10T22:02:40.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:40.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj928 testdata.511250 2024-09-10T22:02:41.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.082 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj929 testdata.511250 2024-09-10T22:02:41.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj930 testdata.511250 2024-09-10T22:02:41.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.375 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj931 testdata.511250 2024-09-10T22:02:41.534 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.535 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj932 testdata.511250 2024-09-10T22:02:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.669 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj933 testdata.511250 2024-09-10T22:02:41.811 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.812 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj934 testdata.511250 2024-09-10T22:02:41.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:41.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj935 testdata.511250 2024-09-10T22:02:42.062 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.063 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj936 testdata.511250 2024-09-10T22:02:42.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.188 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj937 testdata.511250 2024-09-10T22:02:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj938 testdata.511250 2024-09-10T22:02:42.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.473 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj939 testdata.511250 2024-09-10T22:02:42.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj940 testdata.511250 2024-09-10T22:02:42.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.774 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj941 testdata.511250 2024-09-10T22:02:42.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:42.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj942 testdata.511250 2024-09-10T22:02:43.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj943 testdata.511250 2024-09-10T22:02:43.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj944 testdata.511250 2024-09-10T22:02:43.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj945 testdata.511250 2024-09-10T22:02:43.446 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj946 testdata.511250 2024-09-10T22:02:43.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj947 testdata.511250 2024-09-10T22:02:43.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.739 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj948 testdata.511250 2024-09-10T22:02:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:43.931 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj949 testdata.511250 2024-09-10T22:02:44.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj950 testdata.511250 2024-09-10T22:02:44.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.207 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj951 testdata.511250 2024-09-10T22:02:44.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj952 testdata.511250 2024-09-10T22:02:44.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.467 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj953 testdata.511250 2024-09-10T22:02:44.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj954 testdata.511250 2024-09-10T22:02:44.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj955 testdata.511250 2024-09-10T22:02:44.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj956 testdata.511250 2024-09-10T22:02:44.986 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:44.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj957 testdata.511250 2024-09-10T22:02:45.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj958 testdata.511250 2024-09-10T22:02:45.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.246 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj959 testdata.511250 2024-09-10T22:02:45.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj960 testdata.511250 2024-09-10T22:02:45.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.540 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj961 testdata.511250 2024-09-10T22:02:45.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj962 testdata.511250 2024-09-10T22:02:45.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj963 testdata.511250 2024-09-10T22:02:45.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:45.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj964 testdata.511250 2024-09-10T22:02:46.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj965 testdata.511250 2024-09-10T22:02:46.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj966 testdata.511250 2024-09-10T22:02:46.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj967 testdata.511250 2024-09-10T22:02:46.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj968 testdata.511250 2024-09-10T22:02:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.628 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj969 testdata.511250 2024-09-10T22:02:46.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.754 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj970 testdata.511250 2024-09-10T22:02:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:46.880 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj971 testdata.511250 2024-09-10T22:02:47.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.014 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj972 testdata.511250 2024-09-10T22:02:47.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.156 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj973 testdata.511250 2024-09-10T22:02:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.341 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj974 testdata.511250 2024-09-10T22:02:47.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.492 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj975 testdata.511250 2024-09-10T22:02:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj976 testdata.511250 2024-09-10T22:02:47.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj977 testdata.511250 2024-09-10T22:02:47.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:47.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj978 testdata.511250 2024-09-10T22:02:48.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj979 testdata.511250 2024-09-10T22:02:48.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.187 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj980 testdata.511250 2024-09-10T22:02:48.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj981 testdata.511250 2024-09-10T22:02:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj982 testdata.511250 2024-09-10T22:02:48.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj983 testdata.511250 2024-09-10T22:02:48.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.732 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj984 testdata.511250 2024-09-10T22:02:48.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj985 testdata.511250 2024-09-10T22:02:48.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:48.992 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj986 testdata.511250 2024-09-10T22:02:49.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj987 testdata.511250 2024-09-10T22:02:49.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj988 testdata.511250 2024-09-10T22:02:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj989 testdata.511250 2024-09-10T22:02:49.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj990 testdata.511250 2024-09-10T22:02:49.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj991 testdata.511250 2024-09-10T22:02:49.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj992 testdata.511250 2024-09-10T22:02:49.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:49.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj993 testdata.511250 2024-09-10T22:02:50.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj994 testdata.511250 2024-09-10T22:02:50.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj995 testdata.511250 2024-09-10T22:02:50.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj996 testdata.511250 2024-09-10T22:02:50.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj997 testdata.511250 2024-09-10T22:02:50.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj998 testdata.511250 2024-09-10T22:02:50.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj999 testdata.511250 2024-09-10T22:02:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-10T22:02:50.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1000 testdata.511250 2024-09-10T22:02:51.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: rm -f testdata.511250 2024-09-10T22:02:51.037 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: get_primary test obj1 2024-09-10T22:02:51.037 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T22:02:51.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T22:02:51.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T22:02:51.038 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T22:02:51.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: local primary=1 2024-09-10T22:02:51.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: local pgid=1.0 2024-09-10T22:02:51.321 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-09-10T22:02:51.415 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:02:51.415 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:02:51.415 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T22:02:51.415 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-08-27T22:01:11.414313+0000" 2024-09-10T22:02:51.415 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:02:51.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: set -o pipefail 2024-09-10T22:02:51.424 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: found=no 2024-09-10T22:02:51.424 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: seq 0 200 2024-09-10T22:02:51.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:02:51.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-10T22:02:51.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:02:51.425 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:02:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:02:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:02:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:02:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:02:51.697 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:51.698 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:02:51.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-10T22:02:51.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-10T22:02:51.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-10T22:02:51.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:51.798 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:02:51.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672995 2024-09-10T22:02:51.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672995 2024-09-10T22:02:51.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672995' 2024-09-10T22:02:51.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:51.899 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:02:51.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509473 2024-09-10T22:02:51.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509473 2024-09-10T22:02:51.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-64424509473' 2024-09-10T22:02:51.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:52.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-10T22:02:52.000 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:52.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:02:52.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-10T22:02:52.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:52.002 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-10T22:02:52.002 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836517 2024-09-10T22:02:52.003 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-10T22:02:52.003 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:52.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-10T22:02:52.274 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:02:53.275 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:02:53.276 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:53.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836517 2024-09-10T22:02:53.589 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:53.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672995 2024-09-10T22:02:53.590 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:53.590 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:02:53.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672995 2024-09-10T22:02:53.591 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:53.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672995 2024-09-10T22:02:53.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-09-10T22:02:53.592 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672995 2024-09-10T22:02:53.593 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:02:53.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672995 -lt 42949672995 2024-09-10T22:02:53.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:53.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509473 2024-09-10T22:02:53.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:53.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:02:53.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509473 2024-09-10T22:02:53.867 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:53.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509473 2024-09-10T22:02:53.868 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-10T22:02:53.868 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509473 2024-09-10T22:02:53.869 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:02:54.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-10T22:02:54.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-10T22:02:54.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-10T22:02:54.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-10T22:02:54.402 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:02:54.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:02:54.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-10T22:02:54.414 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:02:54.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:02:54.687 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:02:54.687 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:02:54.688 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:02:54.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:02:54.688 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:54.688 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:02:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836519 2024-09-10T22:02:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836519 2024-09-10T22:02:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519' 2024-09-10T22:02:54.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:54.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:02:54.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672997 2024-09-10T22:02:54.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672997 2024-09-10T22:02:54.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672997' 2024-09-10T22:02:54.887 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:54.888 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:02:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509474 2024-09-10T22:02:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509474 2024-09-10T22:02:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672997 2-64424509474' 2024-09-10T22:02:54.988 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:54.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836519 2024-09-10T22:02:54.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:54.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:02:54.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836519 2024-09-10T22:02:54.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:54.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836519 2024-09-10T22:02:54.991 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-10T22:02:54.991 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836519 2024-09-10T22:02:54.991 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:55.261 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836519 2024-09-10T22:02:55.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:02:56.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:02:56.263 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:56.544 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-10T22:02:56.545 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:56.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672997 2024-09-10T22:02:56.545 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:56.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:02:56.546 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672997 2024-09-10T22:02:56.547 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:56.547 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672997 2024-09-10T22:02:56.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672997 2024-09-10T22:02:56.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-10T22:02:56.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:02:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672997 -lt 42949672997 2024-09-10T22:02:56.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:56.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509474 2024-09-10T22:02:56.843 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:56.844 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:02:56.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509474 2024-09-10T22:02:56.845 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509474 2024-09-10T22:02:56.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-10T22:02:56.846 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509474 2024-09-10T22:02:56.846 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:02:57.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-10T22:02:57.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-10T22:02:57.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-10T22:02:57.120 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-10T22:02:57.380 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:02:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:02:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-10T22:02:57.392 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:02:57.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:02:57.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:02:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:02:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:02:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:02:57.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:57.664 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:02:57.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-10T22:02:57.764 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-10T22:02:57.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-10T22:02:57.765 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:57.765 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:02:57.865 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672998 2024-09-10T22:02:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672998 2024-09-10T22:02:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672998' 2024-09-10T22:02:57.866 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:02:57.866 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:02:57.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509476 2024-09-10T22:02:57.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509476 2024-09-10T22:02:57.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672998 2-64424509476' 2024-09-10T22:02:57.965 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:57.965 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-10T22:02:57.966 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:57.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:02:57.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-10T22:02:57.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:57.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-10T22:02:57.968 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-10T22:02:57.968 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836521 2024-09-10T22:02:57.969 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:58.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-10T22:02:58.239 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:02:59.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:02:59.240 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:02:59.527 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-10T22:02:59.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:59.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672998 2024-09-10T22:02:59.528 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:59.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:02:59.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672998 2024-09-10T22:02:59.529 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:59.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672998 2024-09-10T22:02:59.531 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-10T22:02:59.531 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672998 2024-09-10T22:02:59.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:02:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-10T22:02:59.805 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:02:59.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509476 2024-09-10T22:02:59.806 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:02:59.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:02:59.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509476 2024-09-10T22:02:59.807 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:02:59.808 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509476 2024-09-10T22:02:59.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509476 2024-09-10T22:02:59.808 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-10T22:02:59.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:03:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509476 -lt 64424509476 2024-09-10T22:03:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-10T22:03:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-10T22:03:00.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-10T22:03:00.347 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:03:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:03:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-10T22:03:00.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:03:00.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:00.631 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:03:00.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836522 2024-09-10T22:03:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836522 2024-09-10T22:03:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522' 2024-09-10T22:03:00.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:00.731 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:03:00.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673000 2024-09-10T22:03:00.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673000 2024-09-10T22:03:00.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-09-10T22:03:00.832 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:00.833 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:03:00.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509477 2024-09-10T22:03:00.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509477 2024-09-10T22:03:00.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509477' 2024-09-10T22:03:00.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:00.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836522 2024-09-10T22:03:00.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:00.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:03:00.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836522 2024-09-10T22:03:00.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:00.935 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836522 2024-09-10T22:03:00.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836522 2024-09-10T22:03:00.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-10T22:03:00.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:01.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836522 2024-09-10T22:03:01.204 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:03:02.205 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:03:02.205 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:02.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-10T22:03:02.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:02.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673000 2024-09-10T22:03:02.485 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:02.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:03:02.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673000 2024-09-10T22:03:02.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:02.488 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673000 2024-09-10T22:03:02.488 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673000 2024-09-10T22:03:02.489 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-10T22:03:02.489 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:03:02.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673000 -lt 42949673000 2024-09-10T22:03:02.760 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:02.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509477 2024-09-10T22:03:02.761 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:02.761 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:03:02.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509477 2024-09-10T22:03:02.762 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:02.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509477 2024-09-10T22:03:02.763 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-09-10T22:03:02.764 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509477 2024-09-10T22:03:02.764 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:03:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509477 -lt 64424509477 2024-09-10T22:03:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-10T22:03:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-10T22:03:03.055 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-10T22:03:03.322 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:03:03.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:03:03.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-10T22:03:03.334 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:03:03.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:03.646 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:03:03.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-10T22:03:03.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-10T22:03:03.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-10T22:03:03.747 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:03.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:03:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673001 2024-09-10T22:03:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673001 2024-09-10T22:03:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673001' 2024-09-10T22:03:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:03.847 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:03:03.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509479 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509479 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673001 2-64424509479' 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:03.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:03:03.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-10T22:03:03.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:03.950 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-10T22:03:03.950 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836524 2024-09-10T22:03:03.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-10T22:03:03.951 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-10T22:03:04.221 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:03:05.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:03:05.223 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:05.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-10T22:03:05.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:05.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673001 2024-09-10T22:03:05.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:05.522 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:03:05.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673001 2024-09-10T22:03:05.522 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:05.523 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673001 2024-09-10T22:03:05.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673001 2024-09-10T22:03:05.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-10T22:03:05.524 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:03:05.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673002 -lt 42949673001 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509479 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509479 2024-09-10T22:03:05.817 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:05.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509479 2024-09-10T22:03:05.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-10T22:03:05.818 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509479 2024-09-10T22:03:05.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:03:06.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-10T22:03:06.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-10T22:03:06.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-10T22:03:06.093 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-10T22:03:06.352 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:03:06.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: found=yes 2024-09-10T22:03:06.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: break 2024-09-10T22:03:06.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: set +o pipefail 2024-09-10T22:03:06.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: test yes = no 2024-09-10T22:03:06.364 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: ceph osd set nodeep-scrub 2024-09-10T22:03:06.731 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is set 2024-09-10T22:03:06.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-10T22:03:06.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: ceph osd set noscrub 2024-09-10T22:03:07.141 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T22:03:07.153 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:376: _scrub_abort: set -o pipefail 2024-09-10T22:03:07.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: seq 0 200 2024-09-10T22:03:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-10T22:03:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-10T22:03:07.154 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T22:03:07.154 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:07.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:03:07.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836525 2024-09-10T22:03:07.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836525 2024-09-10T22:03:07.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525' 2024-09-10T22:03:07.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:07.571 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:03:07.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673003 2024-09-10T22:03:07.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673003 2024-09-10T22:03:07.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-09-10T22:03:07.680 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:03:07.681 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:03:07.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509481 2024-09-10T22:03:07.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509481 2024-09-10T22:03:07.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509481' 2024-09-10T22:03:07.782 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:07.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836525 2024-09-10T22:03:07.783 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:07.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:03:07.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836525 2024-09-10T22:03:07.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:07.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836525 2024-09-10T22:03:07.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-10T22:03:07.785 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836525 2024-09-10T22:03:07.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836525 2024-09-10T22:03:08.056 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:03:09.057 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T22:03:09.057 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:09.337 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836525 2024-09-10T22:03:09.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T22:03:10.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-10T22:03:10.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:03:10.621 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836525 2024-09-10T22:03:10.622 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:10.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673003 2024-09-10T22:03:10.622 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:10.623 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:03:10.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673003 2024-09-10T22:03:10.623 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:10.624 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673003 2024-09-10T22:03:10.625 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-10T22:03:10.625 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949673003 2024-09-10T22:03:10.625 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:03:10.928 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673003 -lt 42949673003 2024-09-10T22:03:10.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:03:10.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509481 2024-09-10T22:03:10.930 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:03:10.930 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:03:10.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509481 2024-09-10T22:03:10.931 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:03:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509481 2024-09-10T22:03:10.932 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509481 2024-09-10T22:03:10.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-09-10T22:03:10.933 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:03:11.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509481 -lt 64424509481 2024-09-10T22:03:11.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-10T22:03:11.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-10T22:03:11.214 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-10T22:03:11.473 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:03:11.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: break 2024-09-10T22:03:11.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: set +o pipefail 2024-09-10T22:03:11.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: sleep 5 2024-09-10T22:03:16.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-10T22:03:16.500 INFO:tasks.workunit.client.0.smithi181.stdout:2024-09-10T22:03:07.685+0000 7f402d930640 10 osd.1 pg_epoch: 20 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] ] scrubber: nodeep_scrub set, aborting 2024-09-10T22:03:16.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-10T22:03:16.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:16.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:16.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:16.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:16.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: local last_scrub=2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:16.781 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-10T22:03:17.099 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: ceph osd unset nodeep-scrub 2024-09-10T22:03:17.521 INFO:tasks.workunit.client.0.smithi181.stderr:nodeep-scrub is unset 2024-09-10T22:03:17.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-10T22:03:17.533 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: ceph osd unset noscrub 2024-09-10T22:03:18.040 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is unset 2024-09-10T22:03:18.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: TIMEOUT=500 2024-09-10T22:03:18.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:406: _scrub_abort: wait_for_scrub 1.0 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:18.053 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:18.054 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:18.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:18.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:19.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:19.324 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:19.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:19.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:19.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:19.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:19.325 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:19.601 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:20.602 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:20.603 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:20.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:20.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:20.603 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:20.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:20.604 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:20.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:20.881 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:21.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:21.882 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:21.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:21.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:21.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:21.883 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:21.884 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:22.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:22.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:23.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:23.163 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:23.163 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:23.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:23.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:23.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:23.164 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:23.447 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:24.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:24.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:24.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:24.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:24.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:24.449 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:24.729 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:24.730 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:25.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:25.731 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:25.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:25.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:25.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:25.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:25.732 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:26.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:26.011 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:27.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:27.012 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:27.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:27.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:27.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:27.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:27.013 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:27.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:27.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:28.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:28.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:28.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:28.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:28.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:28.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:28.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:28.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:28.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:29.572 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:29.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:29.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:29.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:29.573 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:29.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:29.574 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:29.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:29.852 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:30.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:30.853 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:30.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:30.854 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:30.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:30.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:30.855 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:31.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:31.132 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:32.133 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:32.134 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:32.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:32.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:33.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:33.415 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:33.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:33.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:33.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:33.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:33.416 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:33.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:33.698 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:34.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:34.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:34.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:34.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:34.700 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:34.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:34.701 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:34.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:35.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:35.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:35.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:35.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:35.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:35.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:36.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:36.262 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:37.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:37.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:37.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:37.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:37.264 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:37.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:37.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:37.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:37.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:38.548 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:38.549 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:38.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:38.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:38.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:38.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:38.550 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:38.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:38.833 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:39.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:39.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:39.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:39.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:39.835 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:39.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:39.836 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:40.110 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:40.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:41.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:41.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:41.112 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:41.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:41.113 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:41.390 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:42.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:42.391 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:42.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:42.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:42.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:42.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:42.392 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:42.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:42.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:43.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:43.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:43.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:43.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:43.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:43.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:43.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:43.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:44.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:44.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:44.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:44.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:44.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:44.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:45.222 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:46.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:46.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:46.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:46.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:46.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:46.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:46.225 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:46.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:46.502 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:47.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:47.503 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:47.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:47.504 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:47.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:47.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:47.505 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:47.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:47.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:48.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:48.785 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:48.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:48.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:48.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:48.786 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:48.787 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:49.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:49.064 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:50.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:50.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:50.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:50.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:50.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:50.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:50.066 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:50.347 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:51.348 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:51.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:51.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:51.349 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:51.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:51.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:51.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:51.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:51.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:52.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:52.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:52.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:52.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:52.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:52.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:52.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:52.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:52.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:53.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:53.948 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:53.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:53.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:53.949 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:53.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:53.950 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:54.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:54.230 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:55.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:55.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:55.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:55.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:55.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:55.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:55.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:55.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:55.524 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:56.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:56.525 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:56.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:56.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:56.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:56.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:56.526 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:56.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:56.806 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:57.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:57.807 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:57.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:57.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:57.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:57.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:57.808 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:58.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:58.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:03:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:03:59.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:03:59.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:03:59.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:03:59.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:03:59.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:03:59.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:03:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:03:59.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:00.368 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:00.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:00.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:00.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:00.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:00.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:00.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:00.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:01.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:01.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:01.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:01.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:01.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:01.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:01.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:01.932 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:01.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:02.933 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:02.934 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:02.934 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:02.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:02.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:02.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:02.935 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:03.217 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:03.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:04.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:04.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:04.219 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:04.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:04.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:04.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:04.220 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:04.500 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:05.501 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:05.502 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:05.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:06.783 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:06.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:06.784 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:06.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:06.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:06.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:06.785 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:07.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:07.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:08.065 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:08.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:08.344 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:09.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:09.345 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:09.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:09.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:09.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:09.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:09.346 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:09.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:10.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:10.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:10.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:10.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:10.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:10.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:10.942 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:10.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:11.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:11.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:11.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:11.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:11.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:11.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:12.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:12.218 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:13.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:13.219 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:13.221 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:13.221 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:13.221 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:13.221 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:13.221 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:13.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:13.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:14.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:14.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:14.521 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:14.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:14.803 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:15.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:15.804 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:15.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:15.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:15.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:15.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:15.805 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:16.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:17.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:17.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:17.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:17.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:17.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:17.087 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:17.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:17.366 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:18.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:18.367 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:18.368 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:18.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:18.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:18.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:18.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:18.649 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:18.650 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:19.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:19.651 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:19.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:19.651 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:19.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:19.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:19.652 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:19.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:19.935 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:20.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:20.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:20.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:20.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:20.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:20.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:20.937 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:21.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:21.211 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:22.213 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:22.214 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:22.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:22.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:22.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:22.215 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:22.494 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:23.495 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:23.496 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:23.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:23.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:23.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:23.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:23.497 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:23.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:23.784 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:24.778 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:24.779 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:24.779 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:24.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:24.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:24.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:24.780 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:25.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:26.059 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:26.060 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:26.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:26.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:26.060 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:26.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:26.061 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:26.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:26.338 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:27.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:27.339 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:27.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:27.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:27.339 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:27.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:27.340 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:27.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:28.614 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:28.615 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:28.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:28.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:28.615 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:28.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:28.616 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:28.892 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:29.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:29.893 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:29.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:29.893 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:29.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:29.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:29.894 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:30.173 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:31.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:31.174 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:31.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:31.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:31.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:31.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:31.175 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:31.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:31.452 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:32.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:32.453 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:32.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:32.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:32.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:32.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:32.454 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:32.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:32.724 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:33.725 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:33.726 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:33.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:33.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:33.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:33.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:33.727 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:33.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:33.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:34.997 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:34.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:34.998 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:34.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:34.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:34.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:34.999 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:35.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:35.279 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:36.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:36.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:36.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:36.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:36.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:36.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:36.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:36.564 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:37.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:37.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:37.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:37.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:37.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:37.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:37.567 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:37.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:37.846 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:38.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:38.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:38.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:38.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:38.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:38.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:38.848 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:39.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:39.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:40.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:40.127 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:40.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:40.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:40.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:40.128 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:40.129 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:40.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:40.408 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:41.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:41.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:41.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:41.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:41.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:41.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:41.410 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:41.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:41.690 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:42.691 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:42.692 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:42.692 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:42.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:42.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:42.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:42.693 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:42.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:42.972 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:43.973 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:43.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:43.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:43.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:43.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:43.974 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:44.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:44.253 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:45.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:45.254 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:45.254 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:45.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:45.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:45.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:45.255 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:45.536 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:46.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:46.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:46.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:46.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:46.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:46.538 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:46.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:46.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:47.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:47.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:47.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:47.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:47.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:47.821 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:47.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:48.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:48.101 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:49.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:49.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:49.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:49.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:49.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:49.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:49.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:49.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:49.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:50.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:50.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:50.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:50.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:50.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:50.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:50.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:50.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:50.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:51.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:51.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:51.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:51.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:51.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:51.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:51.666 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:51.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:51.943 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:52.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:52.944 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:52.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:52.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:52.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:52.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:52.945 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:53.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:53.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:54.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:54.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:54.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:54.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:54.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:54.227 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:54.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:54.506 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:55.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:55.507 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:55.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:55.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:55.508 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:55.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:55.509 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:55.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:55.786 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:56.787 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:56.788 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:56.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:56.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:56.789 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:56.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:56.790 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:57.069 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:58.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:58.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:58.071 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:58.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:58.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:58.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:58.072 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:58.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:58.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:04:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:04:59.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:04:59.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:04:59.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:04:59.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:04:59.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:04:59.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:04:59.674 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:04:59.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:00.675 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:00.676 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:00.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:00.676 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:00.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:00.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:00.677 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:00.955 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:00.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:01.956 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:01.957 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:01.957 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:01.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:01.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:01.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:01.958 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:02.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:02.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:03.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:03.237 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:03.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:03.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:03.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:03.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:03.238 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:03.562 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:04.563 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:04.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:04.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:04.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:04.564 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:04.565 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:04.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:04.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:05.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:05.843 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:05.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:05.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:05.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:05.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:05.844 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:06.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:06.125 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:07.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:07.126 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:07.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:07.126 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:07.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:07.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:07.127 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:07.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:07.412 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:08.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:08.413 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:08.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:08.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:08.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:08.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:08.414 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:08.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:08.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:09.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:09.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:09.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:09.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:09.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:09.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:09.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:09.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:09.966 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:10.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:10.967 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:10.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:10.967 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:10.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:10.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:10.968 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:12.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:12.250 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:12.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:12.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:12.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:12.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:12.251 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:12.528 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:12.529 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:13.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:13.530 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:13.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:13.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:13.530 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:13.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:13.531 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:13.815 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:13.816 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:14.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:14.817 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:14.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:14.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:14.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:14.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:14.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:15.095 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:15.096 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:16.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:16.097 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:16.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:16.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:16.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:16.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:16.098 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:16.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:17.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:17.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:17.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:17.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:17.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:17.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:17.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:17.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:17.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:18.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:18.664 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:18.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:18.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:18.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:18.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:18.665 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:18.951 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:19.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:19.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:19.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:19.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:19.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:19.953 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:20.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:20.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:21.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:21.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:21.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:21.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:21.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:21.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:21.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:21.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:21.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:22.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:22.539 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:22.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:22.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:22.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:22.540 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:22.541 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:22.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:22.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:23.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:23.821 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:23.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:23.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:23.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:23.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:23.822 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:24.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:24.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:25.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:25.106 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:25.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:25.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:25.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:25.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:25.107 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:25.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:25.388 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:26.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:26.389 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:26.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:26.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:26.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:26.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:26.390 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:26.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:26.670 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:27.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:27.671 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:27.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:27.671 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:27.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:27.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:27.672 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:27.952 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:28.953 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:28.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:28.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:28.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:28.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:28.954 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:29.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:29.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:30.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:30.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:30.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:30.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:30.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:30.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:30.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:31.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:31.518 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:31.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:31.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:31.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:31.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:31.519 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:31.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:31.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:32.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:32.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:32.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:32.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:32.801 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:33.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:33.080 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:34.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:34.081 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:34.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:34.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:34.082 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:34.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:34.083 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:34.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:35.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:35.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:35.359 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:35.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:35.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:35.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:35.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:35.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:35.641 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:36.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:36.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:36.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:36.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:36.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:36.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:36.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:36.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:36.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:37.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:37.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:37.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:37.921 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:37.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:37.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:37.922 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:38.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:38.198 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:39.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:39.199 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:39.200 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:39.200 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:39.200 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:39.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:39.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:39.483 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:39.484 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:40.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:40.486 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:40.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:40.766 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:41.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:41.767 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:41.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:41.768 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:41.769 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:42.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:42.047 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:43.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:43.049 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:43.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:43.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:43.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:43.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:43.050 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:43.330 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:44.331 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:44.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:44.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:44.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:44.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:44.332 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:44.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:44.612 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:45.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:45.613 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:45.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:45.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:45.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:45.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:45.614 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:45.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:45.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:46.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:46.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:46.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:46.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:46.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:46.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:46.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:47.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:47.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:48.175 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:48.176 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:48.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:48.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:48.176 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:48.177 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:48.454 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:48.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:49.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:49.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:49.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:49.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:49.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:49.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:49.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:49.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:49.734 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:50.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:50.735 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:50.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:50.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:50.735 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:50.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:50.736 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:51.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:51.017 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:52.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:52.018 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:52.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:52.018 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:52.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:52.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:52.019 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:52.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:52.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:53.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:53.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:53.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:53.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:53.300 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:53.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:53.301 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:53.585 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:54.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:54.586 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:54.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:54.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:54.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:54.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:54.587 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:54.869 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:54.870 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:55.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:55.871 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:55.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:55.871 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:55.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:55.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:55.872 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:56.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:56.150 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:57.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:57.151 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:57.152 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:57.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:57.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:57.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:57.153 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:57.430 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:57.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:58.431 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:58.432 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:58.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:58.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:58.432 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:58.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:58.433 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:58.715 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:58.716 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:05:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:05:59.717 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:05:59.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:05:59.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:05:59.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:05:59.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:05:59.718 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:05:59.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:05:59.998 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:00.999 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:01.000 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:01.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:01.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:01.001 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:01.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:01.002 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:01.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:01.280 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:02.281 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:02.281 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:02.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:02.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:02.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:02.282 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:02.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:02.566 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:03.565 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:03.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:03.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:03.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:03.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:03.566 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:03.847 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:04.848 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:04.849 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:05.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:05.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:06.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:06.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:06.131 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:06.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:06.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:06.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:06.132 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:06.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:06.409 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:07.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:07.410 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:07.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:07.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:07.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:07.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:07.411 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:07.693 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:07.694 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:08.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:08.695 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:08.695 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:08.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:08.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:08.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:08.696 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:08.974 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:08.975 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:09.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:09.976 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:09.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:09.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:09.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:09.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:09.977 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:10.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:10.247 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:11.248 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:11.249 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:11.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:11.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:11.249 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:11.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:11.250 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:11.546 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:11.560 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:12.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:12.547 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:12.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:12.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:12.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:12.548 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:12.549 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:12.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:12.829 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:13.830 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:13.831 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:13.831 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:13.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:13.832 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:14.111 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:14.112 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:15.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:15.113 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:15.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:15.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:15.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:15.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:15.114 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:15.386 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:16.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:16.387 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:16.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:16.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:16.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:16.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:16.388 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:16.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:16.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:17.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:17.668 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:17.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:17.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:17.668 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:17.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:17.669 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:17.945 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:17.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:18.946 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:18.947 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:18.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:18.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:18.947 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:18.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:18.948 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:19.228 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:20.229 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:20.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:20.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:20.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:20.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:20.230 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:20.508 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:21.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:21.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:21.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:21.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:21.510 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:21.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:21.511 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:21.791 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:22.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:22.792 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:22.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:22.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:22.792 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:22.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:22.793 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:23.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:23.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:24.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:24.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:24.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:24.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:24.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:24.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:24.076 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:24.358 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:25.359 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:25.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:25.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:25.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:25.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:25.360 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:25.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:25.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:26.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:26.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:26.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:26.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:26.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:26.638 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:26.639 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:26.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:26.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:27.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:27.921 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:27.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:27.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:27.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:27.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:27.923 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:28.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:28.200 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:29.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:29.201 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:29.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:29.201 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:29.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:29.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:29.202 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:29.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:29.485 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:30.486 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:30.487 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:30.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:30.768 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:31.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:31.769 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:31.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:31.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:31.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:31.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:31.770 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:32.050 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:33.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:33.051 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:33.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:33.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:33.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:33.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:33.052 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:33.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:33.332 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:34.333 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:34.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:34.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:34.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:34.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:34.334 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:34.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:34.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:35.618 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:35.619 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:35.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:35.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:35.619 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:35.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:35.620 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:35.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:35.899 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:36.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:36.900 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:36.901 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:36.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:36.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:36.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:36.902 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:37.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:37.178 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:38.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:38.180 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:38.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:38.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:38.181 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:38.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:38.182 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:38.459 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:38.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:39.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:39.460 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:39.461 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:39.462 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:39.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:39.742 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:40.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:40.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:40.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:41.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:41.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:42.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:42.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:42.025 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:42.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:42.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:42.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:42.026 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:42.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:42.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:43.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:43.304 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:43.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:43.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:43.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:43.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:43.305 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:43.607 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:44.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:44.608 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:44.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:44.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:44.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:44.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:44.609 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:44.889 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:44.890 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:45.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:45.891 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:45.891 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:45.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:45.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:45.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:45.892 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:46.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:46.171 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:47.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:47.172 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:47.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:47.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:47.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:47.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:47.173 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:47.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:47.455 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:48.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:48.456 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:48.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:48.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:48.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:48.457 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:48.458 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:48.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:48.737 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:49.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:49.738 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:49.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:49.738 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:49.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:49.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:49.739 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:50.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:50.019 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:51.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:51.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:51.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:51.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:51.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:51.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:51.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:51.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:51.300 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:52.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:52.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:52.302 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:52.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:52.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:52.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:52.303 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:52.578 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:53.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:53.579 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:53.580 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:53.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:53.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:54.859 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:54.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:54.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:54.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:54.860 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:54.861 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:55.142 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:55.143 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:56.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:56.144 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:56.144 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:56.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:56.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:56.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:56.145 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:56.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:57.425 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:57.426 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:57.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:57.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:57.426 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:57.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:57.427 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:57.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:57.706 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:58.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:58.707 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:58.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:58.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:58.708 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:58.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:58.709 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:06:58.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:06:58.987 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:06:59.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:06:59.989 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:06:59.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:06:59.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:06:59.989 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:06:59.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:06:59.990 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:00.268 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:01.269 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:01.270 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:01.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:01.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:01.270 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:01.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:01.271 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:01.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:01.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:02.556 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:02.557 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:02.557 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:02.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:02.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:02.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:02.558 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:02.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:02.837 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:03.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:03.838 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:03.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:03.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:03.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:03.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:03.839 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:04.117 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:04.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:05.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:05.119 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:05.119 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:05.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:05.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:05.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:05.120 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:05.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:05.402 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:06.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:06.403 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:06.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:06.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:06.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:06.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:06.404 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:06.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:06.700 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:07.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:07.701 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:07.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:07.702 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:07.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:07.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:07.703 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:07.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:07.982 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:08.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:08.983 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:08.983 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:08.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:08.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:08.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:08.984 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:09.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:09.263 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:10.264 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:10.265 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:10.265 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:10.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:10.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:10.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:10.266 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:10.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-27T22:01:11.414313+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:10.537 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-10T22:07:11.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-10T22:07:11.538 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-10T22:07:11.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-10T22:07:11.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-10T22:07:11.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-10T22:07:11.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-10T22:07:11.539 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-10T22:07:04.670768+0000 '>' 2024-08-27T22:01:11.414313+0000 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:407: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-10T22:07:11.818 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-10T22:07:11.819 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:11.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-10T22:07:11.820 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-10T22:07:11.912 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_ec": [ 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:11.913 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:11.914 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:11.915 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_repl": [ 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:11.916 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.917 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.918 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_ec": [ 2024-09-10T22:07:11.919 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.920 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:11.921 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.922 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_repl": [ 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:11.923 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.924 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.925 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:11.926 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T22:07:11.927 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:11.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:11.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-10T22:07:11.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_ec": [ 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.014 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.015 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.016 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_repl": [ 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.017 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 2, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 2, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 1, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 1, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 206.940910112, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 206.940910112 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 1, 2024-09-10T22:07:12.018 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 1, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 5.001901128, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 5.001901128 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 41, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 2, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 2, 2024-09-10T22:07:12.019 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0.000999981, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0.000499990 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 2 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.020 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_ec": [ 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.021 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.022 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.023 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_repl": [ 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.024 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.025 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.026 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T22:07:12.027 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:12.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:12.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-10T22:07:12.028 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-10T22:07:12.115 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:07:12.115 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_ec": [ 2024-09-10T22:07:12.115 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.115 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.116 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.117 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.118 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_dp_repl": [ 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "deep", 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.119 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.120 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.121 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_ec": [ 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "ec" 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.122 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.123 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.124 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: ], 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "osd_scrub_sh_repl": [ 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: { 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "labels": { 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "level": "shallow", 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "pooltype": "replicated" 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "counters": { 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_started": 0, 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "num_scrubs_past_reservation": 0, 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs": 0, 2024-09-10T22:07:12.125 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_scrubs_elapsed": { 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_scrubs_elapsed": { 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "preemptions": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_selected": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "chunk_busy": 0, 2024-09-10T22:07:12.126 INFO:tasks.workunit.client.0.smithi181.stdout: "locked_object": 0, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "write_blocked_by_scrub": 0, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reservations_completed": 0, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "successful_reservations_elapsed": { 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.127 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_aborted": 0, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_failure": 0, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "reservation_process_skipped": 0, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "failed_reservations_elapsed": { 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "avgcount": 0, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "sum": 0E-9, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "avgtime": 0E-9 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: "replicas_in_reservation": 0 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: } 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout: ] 2024-09-10T22:07:12.128 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:12.129 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-09-10T22:07:12.129 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T22:07:12.130 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T22:07:12.131 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T22:07:12.231 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T22:07:12.231 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T22:07:12.232 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T22:07:12.232 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T22:07:12.233 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T22:07:12.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T22:07:12.234 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T22:07:12.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:07:12.234 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T22:07:12.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T22:07:12.235 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:07:12.235 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T22:07:12.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T22:07:12.236 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-10T22:07:12.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T22:07:12.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.283 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.283 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.511250 2024-09-10T22:07:12.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T22:07:12.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T22:07:12.284 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-09-10T22:07:12.284 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-09-10T22:07:12.285 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-10T22:07:12.285 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-09-10T22:07:12.286 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T22:07:12.287 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T22:07:12.288 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T22:07:12.289 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T22:07:12.289 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T22:07:12.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:07:12.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T22:07:12.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T22:07:12.290 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:07:12.290 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T22:07:12.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T22:07:12.292 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-10T22:07:12.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T22:07:12.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.293 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.293 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.511250 2024-09-10T22:07:12.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T22:07:12.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T22:07:12.294 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-10T22:07:12.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T22:07:12.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.295 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.295 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.511250 2024-09-10T22:07:12.296 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T22:07:12.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T22:07:12.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T22:07:12.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-10T22:07:12.297 INFO:tasks.workunit.client.0.smithi181.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-09-10T22:07:12.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local poolname=test 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local OSDS=3 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local objects=15 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: TESTDATA=testdata.511250 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-10T22:07:12.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:12.733 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-10T22:07:12.917 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T22:07:12.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T22:07:12.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T22:07:12.918 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.511250/ceph-mon.a.asok 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T22:07:12.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.511250/ceph-mon.a.asok config get fsid 2024-09-10T22:07:12.996 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.511250/ceph-mon.a.asok 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T22:07:12.997 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.511250/ceph-mon.a.asok config get mon_host 2024-09-10T22:07:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:536: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-09-10T22:07:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-10T22:07:13.074 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T22:07:13.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T22:07:13.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T22:07:13.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-10T22:07:13.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:13.878 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T22:07:13.879 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T22:07:13.904 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-10T22:07:13.905 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: expr 3 - 1 2024-09-10T22:07:13.906 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: seq 0 2 2024-09-10T22:07:13.909 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T22:07:13.910 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:07:13.911 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T22:07:13.912 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T22:07:13.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-10T22:07:13.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-10T22:07:13.913 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T22:07:13.913 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=33c920db-0e94-485a-86fa-ae9408fde047 2024-09-10T22:07:13.913 INFO:tasks.workunit.client.0.smithi181.stdout:add osd0 33c920db-0e94-485a-86fa-ae9408fde047 2024-09-10T22:07:13.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 33c920db-0e94-485a-86fa-ae9408fde047' 2024-09-10T22:07:13.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T22:07:13.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQARw+BmZfwvNxAAmazLuoeseD55cesgn29kMA== 2024-09-10T22:07:13.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQARw+BmZfwvNxAAmazLuoeseD55cesgn29kMA=="}' 2024-09-10T22:07:13.927 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 33c920db-0e94-485a-86fa-ae9408fde047 -i td/osd-scrub-test/0/new.json 2024-09-10T22:07:14.125 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:07:14.134 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-10T22:07:14.135 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQARw+BmZfwvNxAAmazLuoeseD55cesgn29kMA== --osd-uuid 33c920db-0e94-485a-86fa-ae9408fde047 2024-09-10T22:07:14.168 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:14.166+0000 7fe75d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:14.175 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:14.173+0000 7fe75d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:14.176 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:14.174+0000 7fe75d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:14.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:14.175+0000 7fe75d00d640 -1 bdev(0x55d66e495c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-10T22:07:14.177 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:14.175+0000 7fe75d00d640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-10T22:07:19.665 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-10T22:07:19.666 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T22:07:19.666 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd0 key to auth repository 2024-09-10T22:07:19.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T22:07:19.667 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T22:07:20.025 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.0 2024-09-10T22:07:20.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T22:07:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T22:07:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T22:07:20.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T22:07:20.055 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:20.054+0000 7feeab064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:20.065 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:20.064+0000 7feeab064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:20.067 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:20.065+0000 7feeab064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:20.301 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:20.302 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:07:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T22:07:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:20.303 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T22:07:20.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:21.213 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:21.212+0000 7feeab064640 -1 Falling back to public interface 2024-09-10T22:07:21.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:21.580 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:21.581 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:07:21.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T22:07:21.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:21.581 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T22:07:21.858 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:22.539 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:22.538+0000 7feeab064640 -1 osd.0 0 log_to_monitors true 2024-09-10T22:07:22.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:22.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:22.860 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:07:22.860 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T22:07:22.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:22.861 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T22:07:23.218 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3822043415,v1:127.0.0.1:6803/3822043415] [v2:127.0.0.1:6804/3822043415,v1:127.0.0.1:6805/3822043415] exists,up 33c920db-0e94-485a-86fa-ae9408fde047 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T22:07:23.223 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:23.224 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-10T22:07:23.225 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-10T22:07:23.226 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T22:07:23.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9708573-abee-4f8b-8bbb-514f51c34598 2024-09-10T22:07:23.226 INFO:tasks.workunit.client.0.smithi181.stdout:add osd1 e9708573-abee-4f8b-8bbb-514f51c34598 2024-09-10T22:07:23.226 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e9708573-abee-4f8b-8bbb-514f51c34598' 2024-09-10T22:07:23.226 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T22:07:23.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbw+BmDMozDhAAqBaQxWqJQ0Igppuz86Xhog== 2024-09-10T22:07:23.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbw+BmDMozDhAAqBaQxWqJQ0Igppuz86Xhog=="}' 2024-09-10T22:07:23.240 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9708573-abee-4f8b-8bbb-514f51c34598 -i td/osd-scrub-test/1/new.json 2024-09-10T22:07:23.565 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:07:23.576 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-10T22:07:23.577 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQAbw+BmDMozDhAAqBaQxWqJQ0Igppuz86Xhog== --osd-uuid e9708573-abee-4f8b-8bbb-514f51c34598 2024-09-10T22:07:23.610 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:23.608+0000 7fdf4d833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:23.649 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:23.648+0000 7fdf4d833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:23.650 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:23.649+0000 7fdf4d833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:23.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:23.650+0000 7fdf4d833640 -1 bdev(0x5647069e2800 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-10T22:07:23.651 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:23.650+0000 7fdf4d833640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-10T22:07:29.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-10T22:07:29.567 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T22:07:29.568 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd1 key to auth repository 2024-09-10T22:07:29.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T22:07:29.568 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T22:07:29.928 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.1 2024-09-10T22:07:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T22:07:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T22:07:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T22:07:29.929 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T22:07:29.959 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:29.957+0000 7fa588013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:29.980 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:29.979+0000 7fa588013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:29.981 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:29.980+0000 7fa588013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:30.243 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:30.244 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:07:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T22:07:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:30.245 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:07:30.515 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:31.516 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:31.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:31.517 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:07:31.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T22:07:31.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:31.517 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:07:31.585 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:31.584+0000 7fa588013640 -1 Falling back to public interface 2024-09-10T22:07:31.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:32.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:32.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:32.799 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:07:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T22:07:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:32.800 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:07:33.083 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:33.163 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:33.162+0000 7fa588013640 -1 osd.1 0 log_to_monitors true 2024-09-10T22:07:34.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:34.085 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:34.085 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T22:07:34.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T22:07:34.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:34.086 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T22:07:34.378 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/522984763,v1:127.0.0.1:6811/522984763] [v2:127.0.0.1:6812/522984763,v1:127.0.0.1:6813/522984763] exists,up e9708573-abee-4f8b-8bbb-514f51c34598 2024-09-10T22:07:34.381 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T22:07:34.382 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' 2024-09-10T22:07:34.383 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T22:07:34.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72 2024-09-10T22:07:34.385 INFO:tasks.workunit.client.0.smithi181.stdout:add osd2 c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72 2024-09-10T22:07:34.385 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72' 2024-09-10T22:07:34.385 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T22:07:34.398 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAmw+Bm9RSvFxAAwpvLwe1zrBdxNIwjgA/XKA== 2024-09-10T22:07:34.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAmw+Bm9RSvFxAAwpvLwe1zrBdxNIwjgA/XKA=="}' 2024-09-10T22:07:34.399 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72 -i td/osd-scrub-test/2/new.json 2024-09-10T22:07:34.786 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:07:34.798 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-10T22:07:34.799 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQAmw+Bm9RSvFxAAwpvLwe1zrBdxNIwjgA/XKA== --osd-uuid c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72 2024-09-10T22:07:34.832 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:34.831+0000 7f0694e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:34.862 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:34.861+0000 7f0694e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:34.863 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:34.862+0000 7f0694e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:34.862+0000 7f0694e60640 -1 bdev(0x560ca27d5c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-10T22:07:34.864 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:34.863+0000 7f0694e60640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-10T22:07:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-10T22:07:40.661 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T22:07:40.662 INFO:tasks.workunit.client.0.smithi181.stdout:adding osd2 key to auth repository 2024-09-10T22:07:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T22:07:40.663 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T22:07:41.021 INFO:tasks.workunit.client.0.smithi181.stdout:start osd.2 2024-09-10T22:07:41.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T22:07:41.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bbaf9de7-74f9-45fb-95be-d1ad74dde5cc --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.511250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-10T22:07:41.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T22:07:41.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T22:07:41.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T22:07:41.052 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:41.050+0000 7fea49e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:41.053 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:41.052+0000 7fea49e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:41.054 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:41.053+0000 7fea49e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T22:07:41.298 INFO:tasks.workunit.client.0.smithi181.stdout:0 2024-09-10T22:07:41.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:41.299 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:07:41.569 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:42.399 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:42.398+0000 7fea49e14640 -1 Falling back to public interface 2024-09-10T22:07:42.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:42.570 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:42.570 INFO:tasks.workunit.client.0.smithi181.stdout:1 2024-09-10T22:07:42.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T22:07:42.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:42.571 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:07:42.854 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:43.855 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:43.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:43.856 INFO:tasks.workunit.client.0.smithi181.stdout:2 2024-09-10T22:07:43.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T22:07:43.856 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:43.857 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:07:44.112 INFO:tasks.workunit.client.0.smithi181.stderr:2024-09-10T22:07:44.110+0000 7fea49e14640 -1 osd.2 0 log_to_monitors true 2024-09-10T22:07:44.160 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T22:07:45.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T22:07:45.161 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-10T22:07:45.161 INFO:tasks.workunit.client.0.smithi181.stdout:3 2024-09-10T22:07:45.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T22:07:45.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T22:07:45.162 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T22:07:45.554 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1796395919,v1:127.0.0.1:6819/1796395919] [v2:127.0.0.1:6820/1796395919,v1:127.0.0.1:6821/1796395919] exists,up c90ecf6a-bb03-40e6-92f0-cbcc9aa14b72 2024-09-10T22:07:45.554 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T22:07:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T22:07:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T22:07:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: create_pool test 1 1 2024-09-10T22:07:45.555 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T22:07:45.925 INFO:tasks.workunit.client.0.smithi181.stderr:pool 'test' created 2024-09-10T22:07:45.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T22:07:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: wait_for_clean 2024-09-10T22:07:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T22:07:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T22:07:46.939 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T22:07:46.939 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T22:07:46.940 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T22:07:46.940 INFO:tasks.workunit.client.0.smithi181.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T22:07:46.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T22:07:46.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T22:07:46.940 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-10T22:07:47.036 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr:1 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr:2' 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:07:47.369 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T22:07:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-10T22:07:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-10T22:07:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-10T22:07:47.469 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:07:47.469 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T22:07:47.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672964 2024-09-10T22:07:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672964 2024-09-10T22:07:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-09-10T22:07:47.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T22:07:47.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T22:07:47.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-10T22:07:47.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-10T22:07:47.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-09-10T22:07:47.743 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:07:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-10T22:07:47.744 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:07:47.744 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T22:07:47.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-10T22:07:47.745 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:07:47.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-10T22:07:47.746 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.0 seq 21474836487 2024-09-10T22:07:47.746 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-10T22:07:47.747 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T22:07:48.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-10T22:07:48.072 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:07:48.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672964 2024-09-10T22:07:48.073 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:07:48.073 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T22:07:48.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672964 2024-09-10T22:07:48.074 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:07:48.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672964 2024-09-10T22:07:48.075 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-10T22:07:48.075 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.1 seq 42949672964 2024-09-10T22:07:48.075 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T22:07:48.349 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-10T22:07:48.350 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T22:07:48.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-10T22:07:48.350 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T22:07:48.351 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T22:07:48.351 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-10T22:07:48.352 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T22:07:48.352 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-10T22:07:48.353 INFO:tasks.workunit.client.0.smithi181.stdout:waiting osd.2 seq 64424509442 2024-09-10T22:07:48.353 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-10T22:07:48.353 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T22:07:48.683 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-10T22:07:48.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T22:07:48.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T22:07:48.684 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T22:07:49.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T22:07:49.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T22:07:49.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T22:07:49.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T22:07:49.027 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T22:07:49.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T22:07:49.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T22:07:49.028 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T22:07:49.297 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T22:07:49.297 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T22:07:49.298 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T22:07:49.298 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T22:07:49.642 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T22:07:49.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T22:07:49.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T22:07:49.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: ceph osd dump 2024-09-10T22:07:49.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-09-10T22:07:49.643 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-09-10T22:07:49.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: poolid=1 2024-09-10T22:07:49.918 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:559: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.511250 bs=1032 count=1 2024-09-10T22:07:49.918 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records in 2024-09-10T22:07:49.918 INFO:tasks.workunit.client.0.smithi181.stderr:1+0 records out 2024-09-10T22:07:49.919 INFO:tasks.workunit.client.0.smithi181.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.4972e-05 s, 22.9 MB/s 2024-09-10T22:07:49.919 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: seq 1 15 2024-09-10T22:07:49.919 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:49.920 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.511250 2024-09-10T22:07:50.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.258 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.511250 2024-09-10T22:07:50.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.384 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.511250 2024-09-10T22:07:50.509 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.510 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.511250 2024-09-10T22:07:50.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.643 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.511250 2024-09-10T22:07:50.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.777 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.511250 2024-09-10T22:07:50.936 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:50.937 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.511250 2024-09-10T22:07:51.070 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.071 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.511250 2024-09-10T22:07:51.196 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.197 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.511250 2024-09-10T22:07:51.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.323 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.511250 2024-09-10T22:07:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.448 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.511250 2024-09-10T22:07:51.573 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.574 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.511250 2024-09-10T22:07:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.699 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.511250 2024-09-10T22:07:51.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.826 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.511250 2024-09-10T22:07:51.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-10T22:07:51.960 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.511250 2024-09-10T22:07:52.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: rm -f testdata.511250 2024-09-10T22:07:52.102 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:566: TEST_dump_scrub_schedule: local pgid=1.0 2024-09-10T22:07:52.103 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2024-09-10T22:07:52.104 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: local now_is=2024-09-10T22:07:52.103468592+00:00 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: declare -A expct_starting 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: declare -A sched_data 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-09-10T22:07:52.103468592+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-10T22:07:52.103468592+00:00 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-10T22:07:52.105 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-10T22:07:52.358 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:07:52.379 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:07:52.379 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:07:52.379 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:07:52.379 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 0, 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-11T22:07:45.875568+0000", 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 18, 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 12 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:52.380 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:07:52.481 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:07:52.481 INFO:tasks.workunit.client.0.smithi181.stdout: "active": false, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "must_scrub": false, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "must_deep_scrub": false, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "must_repair": false, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "need_auto": false, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "scrub_reg_stamp": "2024-09-11T22:07:45.875568+0000", 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrub scheduled @ 2024-09-11T22:07:45.875 (2024-09-11T22:07:45.875)" 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:52.482 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:07:52.587 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=19 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=31 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=false 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrub scheduled' 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='2024-09-11T22:07:45.875' 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='2024-09-11T22:07:45.875' 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=0 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-10T22:07:45.875568+0000' 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='0x0' 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=true 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=true 2024-09-10T22:07:52.588 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=0 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-11T22:07:45.875568+0000' 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=18 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=12 2024-09-10T22:07:52.589 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:07:52.591 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:579: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2024-09-10T22:07:52.592 INFO:tasks.workunit.client.0.smithi181.stdout:-- - comparing: 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_is_future expected: true in actual: true 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active expected: false in actual: false 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:580: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-09-10T22:07:52.593 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-09-10T22:07:52.594 INFO:tasks.workunit.client.0.smithi181.stdout:last-scrub --- 0x0 2024-09-10T22:07:52.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:587: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-10T22:07:45.875568+0000 2024-09-10T22:07:52.594 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:588: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-09-10T22:07:52.685 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T22:07:52.685 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:07:52.685 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:52.698 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T22:07:52.698 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:07:52.698 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:52.711 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T22:07:52.711 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:07:52.711 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:52.720 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:589: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-09-10T22:07:52.827 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:07:52.827 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:07:52.827 INFO:tasks.workunit.client.0.smithi181.stdout: "must": true, 2024-09-10T22:07:52.827 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "0.000000" 2024-09-10T22:07:52.827 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:52.836 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: sleep 5 2024-09-10T22:07:57.839 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:595: TEST_dump_scrub_schedule: sched_data=() 2024-09-10T22:07:57.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-09-10T22:07:57.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-09-10T22:07:57.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-10T22:07:45.875568+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-09-10T22:07:57.840 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-10T22:07:45.875568+0000 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-10T22:07:57.841 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-10T22:07:57.842 INFO:tasks.workunit.client.0.smithi181.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-09-10T22:07:45.875568+0000 (10 retries) 2024-09-10T22:07:58.608 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:07:58.620 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-11T22:07:53.195994+0000", 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 19, 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 39 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:58.621 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:07:58.722 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:07:58.723 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:58.724 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=19 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=40 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:07:58.829 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-10T22:07:53.195994+0000' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-11T22:07:53.195994+0000' 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:07:58.830 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=19 2024-09-10T22:07:58.831 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=39 2024-09-10T22:07:58.831 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:07:58.833 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 1 ~ true / 40 / 39 / false / 2024-09-10T22:07:53.195994+0000 / scrubbing %%% query_last_duration_neg query_last_duration 2024-09-10T22:07:58.833 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-10T22:07:58.833 INFO:tasks.workunit.client.0.smithi181.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sched_data=() 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-10T22:07:45.875568+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-09-10T22:07:58.834 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-10T22:07:45.875568+0000 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-10T22:07:58.835 INFO:tasks.workunit.client.0.smithi181.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-09-10T22:07:45.875568+0000 (10 retries) 2024-09-10T22:07:59.599 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-11T22:07:53.195994+0000", 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:07:59.611 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:07:59.612 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 19, 2024-09-10T22:07:59.612 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 39 2024-09-10T22:07:59.612 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:07:59.612 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:07:59.714 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:07:59.714 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:07:59.714 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:07:59.715 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:07:59.716 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:07:59.717 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=19 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=41 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:07:59.820 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-10T22:07:53.195994+0000' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-11T22:07:53.195994+0000' 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:07:59.821 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:07:59.822 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=19 2024-09-10T22:07:59.822 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=39 2024-09-10T22:07:59.822 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:07:59.823 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 1 ~ true / 41 / 39 / false / 2024-09-10T22:07:53.195994+0000 / scrubbing %%% dmp_last_duration dmp_last_duration_neg 2024-09-10T22:07:59.823 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-10T22:07:59.824 INFO:tasks.workunit.client.0.smithi181.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-10T22:07:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-10T22:07:59.824 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:603: TEST_dump_scrub_schedule: sleep 2 2024-09-10T22:08:01.825 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-10T22:08:01.941 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T22:08:01.941 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-10T22:08:01.941 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:01.956 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T22:08:01.956 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-10T22:08:01.956 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:01.969 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T22:08:01.969 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-10T22:08:01.969 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:01.978 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2024-09-10T22:08:02.083 INFO:tasks.workunit.client.0.smithi181.stdout:osd.0: { 2024-09-10T22:08:02.083 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:08:02.083 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:02.097 INFO:tasks.workunit.client.0.smithi181.stdout:osd.1: { 2024-09-10T22:08:02.097 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:08:02.097 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:02.109 INFO:tasks.workunit.client.0.smithi181.stdout:osd.2: { 2024-09-10T22:08:02.109 INFO:tasks.workunit.client.0.smithi181.stdout: "success": "osd_scrub_sleep = '' " 2024-09-10T22:08:02.109 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:02.118 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:612: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-09-10T22:08:02.508 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is set 2024-09-10T22:08:02.519 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: sleep 2 2024-09-10T22:08:04.520 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-10T22:07:53.195994+0000 2024-09-10T22:08:04.521 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-09-10T22:08:04.624 INFO:tasks.workunit.client.0.smithi181.stdout:{ 2024-09-10T22:08:04.624 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": false, 2024-09-10T22:08:04.625 INFO:tasks.workunit.client.0.smithi181.stdout: "must": false, 2024-09-10T22:08:04.625 INFO:tasks.workunit.client.0.smithi181.stdout: "stamp": "2024-09-03T22:06:24.624104+0000" 2024-09-10T22:08:04.625 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:04.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 1 2024-09-10T22:08:05.636 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: sched_data=() 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-10T22:07:53.195994+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-10T22:07:53.195994+0000 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-10T22:08:05.637 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-10T22:08:05.638 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-10T22:08:05.638 INFO:tasks.workunit.client.0.smithi181.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-09-10T22:07:53.195994+0000 (10 retries) 2024-09-10T22:08:06.409 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:06.421 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:06.421 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:06.421 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:06.421 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:06.421 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "queued for scrub", 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "0", 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": false, 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": false, 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 20, 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 45 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:06.422 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:06.524 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:06.525 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=20 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=46 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:06.630 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='queued for scrub' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='0' 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=false 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=false 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=20 2024-09-10T22:08:06.631 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=45 2024-09-10T22:08:06.632 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 1 ~ true / 46 / 45 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% query_is_future 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-03T22:06:24.624104+0000 2024-09-10T22:08:06.634 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-09-10T22:08:07.011 INFO:tasks.workunit.client.0.smithi181.stderr:noscrub is unset 2024-09-10T22:08:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-09-10T22:08:07.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: declare -A cond_active 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: sched_data=() 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-03T22:06:24.624104+0000 cond_active 'WaitingActive ' sched_data 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-03T22:06:24.624104+0000 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-10T22:08:07.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-10T22:08:07.026 INFO:tasks.workunit.client.0.smithi181.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-03T22:06:24.624104+0000 (10 retries) 2024-09-10T22:08:07.785 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:07.797 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "queued for scrub", 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "0", 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": false, 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": false, 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 20, 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 45 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:07.798 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:07.901 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:07.901 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:07.901 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:07.902 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:07.903 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:07.903 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=48 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:08.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='queued for scrub' 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='0' 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=false 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=false 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=20 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=45 2024-09-10T22:08:08.022 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:08.024 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 1 ~ true / 48 / 45 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% query_active 2024-09-10T22:08:08.024 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-09-10T22:08:08.024 INFO:tasks.workunit.client.0.smithi181.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:642: TEST_dump_scrub_schedule: sched_data=() 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-03T22:06:24.624104+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-10T22:08:08.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-03T22:06:24.624104+0000 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-10T22:08:08.027 INFO:tasks.workunit.client.0.smithi181.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-03T22:06:24.624104+0000 (10 retries) 2024-09-10T22:08:08.787 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:08.798 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "queued for scrub", 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "0", 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": false, 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": false, 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 20, 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 45 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:08.799 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:08.902 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:08.903 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=48 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:09.021 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='queued for scrub' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='0' 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=false 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=false 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=20 2024-09-10T22:08:09.022 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=45 2024-09-10T22:08:09.023 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:09.025 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 1 ~ true / 48 / 45 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:09.025 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:09.025 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:09.784 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:09.795 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:09.795 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "queued for scrub", 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "0", 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": false, 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": false, 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 20, 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 45 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:09.796 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:09.898 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:09.898 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:09.899 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:09.900 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:09.900 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=49 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:10.005 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='queued for scrub' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='0' 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=false 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=false 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=20 2024-09-10T22:08:10.006 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=45 2024-09-10T22:08:10.007 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:10.009 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 2 ~ true / 49 / 45 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:10.009 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:10.009 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:10.768 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:10.780 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:10.781 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:10.781 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:10.781 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:10.883 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:10.883 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:10.884 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:10.885 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:10.989 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=49 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:10.990 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:10.991 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:10.993 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 3 ~ true / 49 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:10.993 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:10.993 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:11.757 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:11.769 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:11.769 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:11.769 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:11.769 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:11.769 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:11.770 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:11.872 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:11.873 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=49 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:11.979 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:11.980 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:11.981 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:11.981 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:11.981 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:11.981 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:11.984 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 4 ~ true / 49 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:11.984 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:11.984 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:12.744 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:12.755 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:12.756 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:12.858 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:12.859 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:12.965 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:12.965 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:12.965 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=49 2024-09-10T22:08:12.965 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:12.965 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:12.966 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:12.967 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:12.967 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:12.967 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:12.969 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 5 ~ true / 49 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:12.969 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:12.969 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:13.728 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:13.739 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:13.739 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:13.739 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:13.739 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:13.739 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:13.740 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:13.841 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:13.842 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:13.843 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:13.949 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:13.949 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:13.949 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=49 2024-09-10T22:08:13.949 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:13.949 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:13.950 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:13.951 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:13.951 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:13.951 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:13.951 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:13.953 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 6 ~ true / 49 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:13.953 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:13.953 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:14.709 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:14.720 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:14.720 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:14.720 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:14.721 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:14.822 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:14.822 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:14.823 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:14.824 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:14.824 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:14.824 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:14.824 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:14.824 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=50 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:14.929 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:14.930 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:14.931 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:14.931 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:14.931 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:14.931 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:14.931 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:14.933 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 7 ~ true / 50 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:14.933 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:14.933 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:15.696 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 48 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:15.708 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:15.812 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:15.812 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:15.812 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:15.812 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:15.812 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:15.813 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:15.814 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:15.814 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:15.814 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:15.814 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:15.942 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:15.942 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:15.942 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=50 2024-09-10T22:08:15.942 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:15.943 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:15.944 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:15.944 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:15.944 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=48 2024-09-10T22:08:15.944 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:15.946 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 8 ~ true / 50 / 48 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:15.946 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:15.946 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:16.705 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:16.716 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:16.716 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:16.716 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 49 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:16.717 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:16.819 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:16.820 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:16.925 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=50 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:16.926 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=49 2024-09-10T22:08:16.927 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:16.929 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 9 ~ true / 50 / 49 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:16.929 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:16.929 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:17.687 INFO:tasks.workunit.client.0.smithi181.stderr:dumped pgs 2024-09-10T22:08:17.698 INFO:tasks.workunit.client.0.smithi181.stdout:from pg dump pg: { 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_pg_state": "active+clean", 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_state_has_scrubbing": false, 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_last_duration": 1, 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_schedule_at": "2024-09-10T22:09:05.239126+0000", 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_is_future": true, 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_vs_date": true, 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_reported_epoch": 21, 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout: "dmp_seq": 49 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout:} 2024-09-10T22:08:17.699 INFO:tasks.workunit.client.0.smithi181.stdout:query output: 2024-09-10T22:08:17.801 INFO:tasks.workunit.client.0.smithi181.stdout: "scrubber": { 2024-09-10T22:08:17.801 INFO:tasks.workunit.client.0.smithi181.stdout: "active": true, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "epoch_start": "17", 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "start": "MIN", 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "end": "MIN", 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "max_end": "MIN", 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "subset_last_update": "0'0", 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "deep": true, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "req_scrub": false, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "auto_repair": false, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "check_repair": false, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_scrub_on_error": false, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "priority": 0, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "shallow_errors": 0, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "deep_errors": 0, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "fixed": 0, 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "waiting_on_whom": [], 2024-09-10T22:08:17.802 INFO:tasks.workunit.client.0.smithi181.stdout: "schedule": "scrubbing" 2024-09-10T22:08:17.803 INFO:tasks.workunit.client.0.smithi181.stdout: }, 2024-09-10T22:08:17.803 INFO:tasks.workunit.client.0.smithi181.stdout: "agent_state": {} 2024-09-10T22:08:17.908 INFO:tasks.workunit.client.0.smithi181.stdout:( 2024-09-10T22:08:17.908 INFO:tasks.workunit.client.0.smithi181.stdout:[query_epoch]=21 2024-09-10T22:08:17.908 INFO:tasks.workunit.client.0.smithi181.stdout:[query_seq]=50 2024-09-10T22:08:17.908 INFO:tasks.workunit.client.0.smithi181.stdout:[query_active]=true 2024-09-10T22:08:17.908 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule]='scrubbing' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_schedule_at]='0' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_target_at]='0' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_duration]=1 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_stamp]='2024-09-03T22:06:24.624104+0000' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_last_scrub]='19x15' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_is_future]=false 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_vs_date]=false 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[query_scrub_seq]=null 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_pg_state]='active+clean' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_state_has_scrubbing]=false 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_last_duration]=1 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_schedule_at]='2024-09-10T22:09:05.239126+0000' 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_is_future]=true 2024-09-10T22:08:17.909 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_vs_date]=true 2024-09-10T22:08:17.910 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_reported_epoch]=21 2024-09-10T22:08:17.910 INFO:tasks.workunit.client.0.smithi181.stdout:[dmp_seq]=49 2024-09-10T22:08:17.910 INFO:tasks.workunit.client.0.smithi181.stdout:) 2024-09-10T22:08:17.912 INFO:tasks.workunit.client.0.smithi181.stdout:--> loop: 10 ~ true / 50 / 49 / false / 2024-09-03T22:06:24.624104+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-10T22:08:17.912 INFO:tasks.workunit.client.0.smithi181.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-10T22:08:17.912 INFO:tasks.workunit.client.0.smithi181.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-10T22:08:17.912 INFO:tasks.workunit.client.0.smithi181.stdout:WaitingActive : wait_any_cond(): failure. Note: query-active=true 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:164: wait_any_cond: return 1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_dump_scrub_schedule: return 1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: return 1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-test 1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T22:08:17.914 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T22:08:17.915 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T22:08:18.020 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T22:08:18.020 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T22:08:18.021 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T22:08:18.021 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T22:08:18.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T22:08:18.022 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T22:08:18.022 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T22:08:18.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:08:18.023 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T22:08:18.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T22:08:18.024 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T22:08:18.024 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T22:08:18.025 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-10T22:08:18.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-10T22:08:18.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-10T22:08:18.026 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-scrub-test/mgr.x.log td/osd-scrub-test/mon.a.log td/osd-scrub-test/osd.0.log td/osd-scrub-test/osd.1.log td/osd-scrub-test/osd.2.log /home/ubuntu/cephtest/archive/log 2024-09-10T22:08:18.027 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-10T22:08:18.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T22:08:18.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T22:08:18.031 INFO:tasks.workunit.client.0.smithi181.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.511250 2024-09-10T22:08:18.031 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.511250 2024-09-10T22:08:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T22:08:18.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T22:08:18.032 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T22:08:18.033 INFO:tasks.workunit.client.0.smithi181.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-10T22:08:18.033 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-09-10T22:08:18.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T22:08:18.357 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-09-10T22:08:18.543 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4e4b4b92f3f345e1885cec7e8c51b957 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-09-10T22:08:18.546 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T22:08:18.554 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T22:08:18.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T22:08:18.627 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T22:08:18.635 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T22:08:18.635 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T22:08:18.649 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-10T22:08:18.653 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T22:08:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-10T22:08:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 7 377 70 -363us[ -426us] +/- 42ms 2024-09-10T22:08:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 +1297us[+1297us] +/- 62ms 2024-09-10T22:08:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 9 377 249 -925us[ -993us] +/- 72ms 2024-09-10T22:08:18.654 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T22:08:18.655 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T22:08:18.663 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T22:08:18.663 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T22:08:18.671 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T22:08:18.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T22:08:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T22:08:18.782 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T22:08:18.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T22:08:18.799 INFO:teuthology.task.internal:Duration was 6951.690408 seconds 2024-09-10T22:08:18.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T22:08:18.807 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T22:08:18.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T22:08:18.860 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T22:08:19.119 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T22:08:19.120 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-10T22:08:19.120 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T22:08:19.181 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T22:08:19.181 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T22:08:19.248 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T22:08:19.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T22:08:19.385 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T22:08:19.394 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T22:08:19.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T22:08:19.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T22:08:19.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T22:08:19.482 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-10T22:08:19.508 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T22:08:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T22:08:19.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T22:08:19.574 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T22:08:19.576 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899557/remote/smithi181 2024-09-10T22:08:19.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T22:08:20.160 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T22:08:20.160 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T22:08:20.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T22:08:20.204 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T22:08:20.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T22:08:20.213 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T22:08:20.213 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T22:08:20.251 INFO:teuthology.orchestra.run.smithi181.stdout: 265324 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 22:08 /home/ubuntu/cephtest 2024-09-10T22:08:20.251 INFO:teuthology.orchestra.run.smithi181.stdout: 405775 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 20:18 /home/ubuntu/cephtest/mnt.0 2024-09-10T22:08:20.251 INFO:teuthology.orchestra.run.smithi181.stdout: 405776 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 21:59 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T22:08:20.251 INFO:teuthology.orchestra.run.smithi181.stdout: 405819 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 22:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T22:08:20.251 INFO:teuthology.orchestra.run.smithi181.stdout: 420643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 22:08 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-10T22:08:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T22:08:20.252 INFO:teuthology.orchestra.run.smithi181.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-10T22:08:20.253 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T22:08:20.257 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T22:08:20.267 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T22:08:20.276 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi181 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T22:08:20.279 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 6951.6904084682465 failure_reason: 'Command failed (workunit test scrub/osd-scrub-test.sh) on smithi181 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4e4b4b92f3f345e1885cec7e8c51b957 status: fail success: false 2024-09-10T22:08:20.279 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T22:08:20.432 INFO:teuthology.run:FAIL